Facial recognition authentication system including path parameters

ABSTRACT

Systems and methods for enrolling and authenticating a user in an authentication system via a user&#39;s camera of camera equipped mobile device include capturing and storing enrollment biometric information from at least one first image of the user taken via the camera of the mobile device, capturing authentication biometric information from at least one second image of the user, capturing, during imaging of the at least one second image, path parameters via at least one movement detecting sensor indicating an authentication movement of the mobile device, comparing the authentication biometric information to the stored enrollment biometric information, and comparing the authentication movement of the mobile device to an expected movement of the mobile device to determine whether the authentication movement sufficiently corresponds to the expected movement.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Application No.62/460,670 which was filed on Feb. 17, 2017 and is acontinuation-in-part of U.S. application Ser. No. 14/839,505 which wasfiled on Aug. 28, 2015, which application claims the benefit of U.S.Provisional Application No. 62/043,224 which was filed Aug. 28, 2014;U.S. Provisional Application No. 62/054,847 which was filed on Sep. 24,2014; U.S. Provisional Application No. 62/064,415 which was filed onOct. 15, 2014; U.S. Provisional Application No. 62/085,963 which wasfiled on Dec. 1, 2014; U.S. Provisional Application No. 62/101,317 whichwas filed on Jan. 8, 2015; U.S. Provisional Application No. 62/139,558which was filed on Mar. 27, 2015; and U.S. Provisional Application No.62/188,584 which was filed on Jul. 3, 2015. The contents of each of theabove applications are incorporated by reference.

BACKGROUND 1. Field

The disclosed embodiments relate to biometric security. Morespecifically, the disclosed embodiments relate to facial recognitionauthentication systems.

2. Related Art

With the growth of personal electronic devices that may be used toaccess several user accounts, and the increasing threat of identitytheft and other security issues, there is a growing need for ways tosecurely access user accounts via electronic devices. Account holdersare thus often required to have longer passwords that meet variouscriteria such as using a mixture of capital and lowercase letters,numbers, and other symbols. With smaller electronic devices, such assmart phones, smart watches, “Internet of Things” (“IoT”) devices andthe like, it may become cumbersome to attempt to type such longpasswords into the device each time access to the account is desired. Insome instances, users may even decide to deactivate such cumbersomesecurity measures due to their inconvenience on their devices. Thus,users of such devices may prefer other methods of secure access to theiruser accounts.

One other such method may be using biometrics. For example, anelectronic device may have an optical reader that may scan a user'sfingerprint to determine that the person requesting access to a deviceor an account is authorized. However, such fingerprint systems are oftenprohibitively large and expensive for use on an inexpensive electronicdevice and are often considered unreliable and unsecure.

In addition, facial recognition is generally known and may be used in avariety of contexts. Two-dimensional facial recognition is commonly usedto tag people in images on social networks or in photo editing software.Facial recognition software, however, has not been widely implemented onits own to securely authenticate users attempting to gain access to anaccount because it not considered secure enough. For example,two-dimensional facial recognition is considered unsecure because facesmay be photographed or recorded, and then the resulting prints or videodisplays showing images of the user may be used to spoof the system.Accordingly, there is a need for reliable, cost-effective, andconvenient method to authenticate users attempting to log in to, forexample, a user account.

SUMMARY

The disclosed embodiments have been developed in light of the above andaspects of the invention may include a method for enrolling andauthenticating a user in an authentication system via a user's computingdevice. The user's device includes a camera and, in some instances,includes at least one movement detecting sensor, such as anaccelerometer, magnetometer, and gyroscope, or the camera is used forthis function.

In one embodiment, the user may enroll in the system by providingenrollment images of the user's face. The enrollment images are taken bythe camera of the mobile device as the user moves the mobile device todifferent positions relative to the user's head. The user may thusobtain enrollment images showing the user's face from different anglesand distances. The system may also utilize one or more movement sensorsof a mobile device to determine an enrollment movement path that thephone takes during the imaging. At least one image is processed todetect the user's face within the image, and to obtain biometricinformation from the user's face in the image. The image processing maybe done on the user's mobile device or at a remote device, such as anauthentication server or a user account server. The enrollmentinformation (the enrollment biometrics, movement, and other information)may be stored on the mobile device or remote device.

The system may then authenticate a user by the user providing at leastone authentication image via the camera of the mobile device while theuser moves the mobile device to different positions relative to theuser's head. In other embodiments, multiple authentication images may berequired. In some examples, the user may move his or her face todifferent positions relative to the camera of the computing device ormove the camera to a different position relative to their face. Theauthentication images are processed for face detection and facialbiometric information. Path parameters are also obtained during theimaging of the authentication images (authentication movement). Theauthentication information (authentication biometric, movement, andother information) is then compared with the enrollment information todetermine whether the user should be authenticated or denied. Imageprocessing and comparison may be conducted on the user's mobile device,or may be conducted remotely.

In some embodiments, multiple enrollment profiles may be created by auser to provide further security. For example, a user may create anenrollment wearing accessories such as a hat or glasses, or while makinga funny face. In further embodiments, the user's enrollment informationmay be linked to a user email address, phone number, or otheridentifier.

The authentication system may include feedback displayed on the mobiledevice to aid a user in learning and authentication with the system. Forinstance, an accuracy meter may provide feedback on a match rate of theauthentication biometrics or movement. A movement meter may providefeedback on the movement detected by the mobile device.

In some embodiments, the system may reward users who successfullyutilize the authentication system or who otherwise take fraud preventingmeasures. Such rewards may include leaderboards, status levels, rewardpoints, coupons or other offers, and the like. In some embodiments, theauthentication system may be used to login to multiple accounts.

In addition to biometric and movement matching, some embodiments mayalso utilize banding detection, glare detection, and screen edgedetection to further secure the system. In other embodiments, other userattributes may be detected and matched including users' gender, age,ethnicity, and the like.

The system may also provide gradual access to user account(s) when theuser first sets up the authentication system. As the user successfullyimplements the system, authorization may be expanded. For example,during a time period as the user gets accustomed to the authenticationsystem, lower transaction limits may be applied.

In some embodiments, the mobile device may show video feedback of whatthe user is imaging to aid the user to image his or her face duringenrollment or authentication. The video feedback may be displayed ononly a portion of the display screen of the mobile device. For example,the video feedback may be displayed in an upper portion of the displayscreen. The video feedback display may be position on a portion of thedisplay screen that corresponds with a location of a front-facing cameraof the mobile device.

To facilitate imaging in low-light, portions of the screen other thanthe video feedback may be displayed in a bright color, such as white. Insome embodiments, and LED or infrared light may be used, and nearinfrared thermal imaging may be done with an infrared camera. The mobiledevice used for imaging may thus have multiple cameras for capturevisible light and infrared images. The mobile device may also havemultiple cameras (two or more) imaging in a single spectrum or multiplespectrum to provide stereoscopic, three-dimensional images. In such anembodiment, the close-up frames (zoomed) may create the mostdifferentiation as compared to images captured from a distance. In suchan embodiment, the frames captured at a distance may be unnecessary.

In some embodiments, to provide added security, the mobile device mayoutput objects, colors, or patterns on the display screen to be detectedduring the imaging. The predetermined object or pattern may be a uniqueone-dimensional or two-dimensional barcode. For example, a QR code(two-dimensional barcode) may be displayed on the screen and reflectedoff the user's eye. If the QR code is detected in the image, then theperson may be authenticated. In other embodiments, an object may move onthe screen and the system may detect whether a user's eyes follow themovement.

In some embodiments, the system may provide prompts on a video feedbackdisplay to aid the user in moving the device relative to the user's headduring enrollment and/or authentication. The prompts may include ovalsor frames displayed on the display screen in which the user must placehis or her face by moving the mobile device until his or her face iswithin the oval or frame. The prompts may preferably be of differingsizes and may also be centered on different positions of the screen.When an actual, three-dimensional person images himself or herself closeup and far away, it has been found that the biometric results aredifferent due to perspective distortion as compared to when a spoofattempt is made using two-dimensional pictures that include imagedpictures of the person both close up and far away. Thus, athree-dimensional person may be validated when biometric results showdifferences in perspective distortion in the close-up and far awayimages. This also allows the user to have multiple biometric profilesfor each of the distances.

In other embodiments, biometrics from images obtained between theclose-up and far away images may be analyzed for incrementally differentbiometric results. In this manner, the morphing of the face from the farface to the warped close up face is captured and tracked. Theincremental frames during an authentication may then be matched toframes captured at similar locations during enrollment along the motionpath and compared to ensure that the expected similarities anddifference are found. This results in a motion path and captured imageand biometric data that can prove a three-dimensional face is presentlybeing imaged. Thus, not only are the close-up and far away biometricscompared, but also biometric data obtained in between. The biometricdata obtained in between must also correspond to a correct morphingspeed along the motion path, greatly enhancing the security of thesystem.

The touch screen may be utilized in some embodiments. For example, theuser may need to enter a swipe a particular code or pattern in additionto the authentication system described herein. The touchscreen may alsodetect a size and orientation of a user's finger, and whether a righthand or a left hand is used on the touch screen. Voice parameters mayalso be used as an added layer of security. The system may detect edgesharpness or other indicators to ensure that the obtained images are ofsufficient quality for the authentication system.

When a camera has an autofocus, the autofocus may be controlled by thesystem to validate the presence of the actual, three-dimensional person.The autofocus may check that different features of the user orenvironment focus at different focus points, which also may be referredto as focal lengths. This disclosure covers both focus points and focallength. In other embodiments, authentication images may be saved toreview the person who attempted to authenticate with the system.

In some embodiments, the match thresholds required may be adapted overtime. The system may thus account for changing biometrics due to age,weight gain/loss, environment, user experience, security level, or otherfactors. In further embodiments, the system may utilize image distortionprior to obtaining biometric information to further protect againstfraudulent access.

The system may utilize any number or combination of the securityfeatures as security layers, as described herein. When authenticationfails, the system may be configured so that it is unclear which securitylayer triggered the failure to preserve the integrity of the securitysystem.

Other systems, methods, features and advantages of the invention will beor will become apparent to one with skill in the art upon examination ofthe following figures and detailed description. It is intended that allsuch additional systems, methods, features and advantages be includedwithin this description, be within the scope of the invention, and beprotected by the accompanying claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The components in the figures are not necessarily to scale, emphasisinstead being placed upon illustrating the principles of the invention.In the figures, like reference numerals designate corresponding partsthroughout the different views.

FIG. 1 illustrates an example environment of use of the facialrecognition authentication system, according to one exemplaryembodiment.

FIG. 2 illustrates an example embodiment of a mobile device.

FIG. 3 illustrates exemplary software modules that are part of themobile device and server.

FIG. 4 shows a method for performing facial recognition authenticationaccording to one embodiment.

FIG. 5 shows a method for enrolling a user in a facial recognitionauthentication system, according to one exemplary embodiment.

FIGS. 6A and 6B show an example of movement of a mobile device about auser's face according to one exemplary embodiment.

FIGS. 7A and 7B show an example of movement of a mobile device about auser's face according to one exemplary embodiment.

FIG. 8 shows a method of providing authentication information in afacial recognition authentication system, according to one exemplaryembodiment.

FIG. 9 shows a method of verifying authentication credentials in afacial recognition authentication system, according to one exemplaryembodiment.

FIG. 10 illustrates an exemplary display showing a graphical and numericfeedback in a facial recognition authentication system.

FIGS. 11A, 11B, and 11C illustrate exemplary video feedback displayscorresponding to front-facing camera positions in a facial recognitionauthentication system.

FIG. 12 shows an exemplary video display feedback of a facialrecognition authentication system where edge pixels on the sides of thedisplay are stretched horizontally.

FIGS. 13A and 13B illustrates exemplary screen displays with facealignment indicators shown as an oval to serve as a guide as the usermoves the mobile device closer to or away from their face.

FIG. 14 shows a method of verifying liveness or three-dimensionality ofa user utilizing pixel velocity analysis detection.

FIG. 15 illustrates an exemplary mobile device display showing agraphical code entry interface with an imaging area.

FIG. 16 illustrates an example mobile device display showing a numericand graphical code entry interface with an imaging area.

DETAILED DESCRIPTION OF EMBODIMENTS

A system and method for providing secure and convenient facialrecognition authentication will be described below. The system andmethod may be achieved without the need for additional expensivebiometric readers or systems while offering enhanced security overconventional facial recognition systems.

Facial Recognition Authentication Environment

FIG. 1 illustrates an example environment of use of the facialrecognition authentication system described herein. This is but onepossible environment of use and system. It is contemplated that, afterreading the specification provided below in connection with the figures,one of ordinary skill in the art may arrive at different environments ofuse and configurations.

In this environment, a user 108 may have a mobile device 112 which maybe used to access one or more of the user's accounts via authenticationsystems. A user 108 may have a mobile device 112 that can capture apicture of the user 108, such as an image of the user's face. The usermay use a camera 114 on or connected to the mobile device 112 to capturean image or multiple images or video of himself or herself. The mobiledevice 112 may comprise any type of mobile device capable of capturingan image, either still or video, and performing processing of the imageor communication over a network.

In this embodiment, the user 108 may carry and hold the mobile device112 to capture the image. The user may also wear or hold any number ofother devices. For, example, the user may wear a watch 130 containingone or more cameras 134 or biosensors disposed on the watch. The camera134 may be configured to create an image from visible light as well asinfrared light. The camera 134 may additionally or alternatively employimage intensification, active illumination, or thermal vision to obtainimages in dark environments.

When pointed towards a user 108, the camera 134 may capture an image ofthe user's face. The camera 134 may be part of a module that may eitherinclude communication capability that communicates with either a mobiledevice 112, such as via Bluetooth®, NFC, or other format, orcommunication directly with a network 116 over a wired or wireless link154. The watch 130 may include a screen on its face to allow the user toview information. If the camera module 134 communicates with the mobiledevice 112, the mobile device 134 may relay communications to thenetwork 116. The mobile device 134 may be configured with more than onefront facing camera 114 to provide for a 3D or stereoscopic view, or toobtain images across different spectral ranges, such as near infraredand visible light.

The mobile device 112 is configured to wirelessly communicate over anetwork 116 with a remote server 120. The server 120 may communicatewith one or more databases 124. The network 116 may be any type ofnetwork capable of communicating to and from the mobile device includingbut not limited to a LAN, WAN, PAN, or the Internet. The mobile device112 may communicate with the network via a wired or wireless connection,such as via Ethernet, Wi-Fi, NFC, and the like. The server 120 mayinclude any type of computing device capable of communicating with themobile device 112. The server 120 and mobile device 112 are configuredwith a processor and memory and are configured to execute machinereadable code or machine instructions stored in the memory.

The database 124, stored on mobile device or remote location as shown,may contain facial biometric information and authentication informationof users 108 to identify the users 108 to allow access to associateduser data based on one or more images or biometric information receivedfrom the mobile device 112 or watch 134. The data may be, for example,information relating to a user account or instruction to allow access toa separate account information server 120B. The term biometric data mayinclude among other information biometric information concerning facialfeatures (distorted or undistorted) and path parameters. Examples ofpath parameters may include an acceleration and speed of the mobiledevice, angle of the mobile device during image capture, distance of themobile device to the user, path direction in relation to the user's faceposition in relation to the user, or any other type parameter associatedwith movement of the mobile device or the user face in relation to acamera. Other data may also be included such as GPS data, deviceidentification information, and the like.

In this embodiment, the server 120 processes requests for identificationfrom the mobile device 112 or user 108. In one configuration, the imagecaptured by the mobile device 112, using facial detection, comprises oneor more images of the user's face 108 during movement of the mobiledevice relative to the user's face, such as in a side to side orhorizontal arc or line, vertical arc or line, forward and backwards fromthe user's face, or any other direction of motion. In anotherconfiguration, the mobile device 112 calculates biometric informationfrom the obtained images and sends the biometric information to theserver 120. In yet another embodiment, the mobile device 112 comparesbiometric information with stored biometric information on the mobiledevice 112 and sends an authentication result from the comparison to theserver 120.

The data including either the image(s), biometric information, or bothare sent over the network 116 to the server 120. Using image processingand image recognition algorithms, the server 120 processes the person'sbiometric information, such as facial data, and compares the biometricinformation with biometric data stored in the database 124 to determinethe likelihood of a match. In other embodiments, the image processingand comparison is done on the mobile device 112, and data sent to theserver indicates a result of the comparison. In further embodiments, theimage processing and comparison is done on the mobile device 112 withoutaccessing the server, for example, to obtain access to the mobile device112 itself.

By using facial recognition processing, an accurate identity match maybe established. Based on this and optionally one or more other factors,access may be granted, or an unauthorized user may be rejected. Facialrecognition processing is known in the art (or is an establishedprocess) and as a result, it is not described in detail herein.

Also shown is a second server 120B with associated second database 124B,and third server 120C with associated third database 124C. The secondand third database may be provided to contain additional informationthat is not available on the server 120 and database 124. For example,one of the additional servers may only be accessed based on theauthentication of the user 108 performed by the server 120.

Executing on the mobile device 112 is one or more software applications.This software is defined herein as an identification application (IDApp). The ID App may be configured with either or both of facialdetection and facial recognition and one or more software modules whichmonitor the path parameters and/or biometric data. Facial detection asused herein refers to a process which detects a face in an image. Facialrecognition as used herein refers to a process that can analyze a faceusing an algorithm, mapping its facial features, and converting them tobiometric data, such as numeric data. The biometric data can be comparedto that derived from one or more different images for similarities ordis-similarities. If a high percentage of similarity is found in thebiometric data, the individual shown in the images may be considered tobe a match.

With the ultimate goal of matching a face of a user to an identity orimage stored in a database 124, to authenticate the user, the ID App mayfirst process the image captured by the camera 114, 134 to identify andlocate the face that is in the image. As shown in FIG. 1, there may bethe face 108. The authentication may be used for logging into an onlineaccount or for numerous other access control functions.

The portion of the photo that contains the detected face may then becropped, cut, and stored for processing by one or more facialrecognition algorithms. By first detecting the face in the image andcropping only that portion of the face, the facial recognition algorithmneed not process the entire image. Further, in embodiments where thefacial recognition processing occurs remotely from the mobile device112, such as at a server 120, much less image data is required to besent over the network to the remote location. It is contemplated thatthe entire image, a cropped face, or only biometric data may be sent tothe remote server 120 for processing.

Facial detection software can detect a face from a variety of angles.However, facial recognition algorithms are most accurate in straight onimages in well-lit situations. In one embodiment, the highest qualityface image for facial recognition that is captured is processed first,then images of the face that are lower quality or at different anglesother than straight toward the face are then processed. The processingmay occur on the mobile device or at a remote server which has access tolarge databases of image data or facial identification data.

The facial detection is preferred to occur on the mobile device and isperformed by the mobile device software, such as the ID App. Thisreduces the number or size of images (data) that are sent to the serverfor processing where faces are not found and minimizes the overallamount of data that must be sent over the network. This reducesbandwidth needs and network speed requirements are reduced.

In another preferred embodiment, the facial detection, facialrecognition, and biometric comparison all occur on the mobile device.However, it is contemplated that the facial recognition processing mayoccur on the mobile device, the remote server, or both.

FIG. 2 illustrates an example embodiment of a mobile device. This is butone possible mobile device configuration and as such it is contemplatedthat one of ordinary skill in the art may differently configure themobile device. The mobile device 200 may comprise any type of mobilecommunication device capable of performing as described below. Themobile device may comprise a PDA, cellular telephone, smart phone,tablet PC, wireless electronic pad, an IoT device, a “wearable”electronic device or any other computing device.

In this example embodiment, the mobile device 200 is configured with anouter housing 204 configured to protect and contain the componentsdescribed below. Within the housing 204 is a processor 208 and a firstand second bus 212A, 212B (collectively 212). The processor 208communicates over the buses 212 with the other components of the mobiledevice 200. The processor 208 may comprise any type processor orcontroller capable of performing as described herein. The processor 208may comprise a general-purpose processor, ASIC, ARM, DSP, controller, orany other type processing device. The processor 208 and other elementsof the mobile device 200 receive power from a battery 220 or other powersource. An electrical interface 224 provides one or more electricalports to electrically interface with the mobile device, such as with asecond electronic device, computer, a medical device, or a powersupply/charging device. The interface 224 may comprise any typeelectrical interface or connector format.

One or more memories 210 are part of the mobile device 200 for storageof machine readable code for execution on the processor 208 and forstorage of data, such as image data, audio data, user data, medicaldata, location data, accelerometer data, or any other type of data. Thememory 210 may comprise RAM, ROM, flash memory, optical memory, ormicro-drive memory. The machine-readable code as described herein isnon-transitory.

As part of this embodiment, the processor 208 connects to a userinterface 216. The user interface 216 may comprise any system or deviceconfigured to accept user input to control the mobile device. The userinterface 216 may comprise one or more of the following: keyboard,roller ball, buttons, wheels, pointer key, touch pad, and touch screen.A touch screen controller 230 is also provided which interfaces throughthe bus 212 and connects to a display 228.

The display comprises any type display screen configured to displayvisual information to the user. The screen may comprise a LED, LCD, thinfilm transistor screen, OEL CSTN (color super twisted nematic), TFT(thin film transistor), TFD (thin film diode), OLED (organiclight-emitting diode), AMOLED display (active-matrix organiclight-emitting diode), capacitive touch screen, resistive touch screenor any combination of these technologies. The display 228 receivessignals from the processor 208 and these signals are translated by thedisplay into text and images as is understood in the art. The display228 may further comprise a display processor (not shown) or controllerthat interfaces with the processor 208. The touch screen controller 230may comprise a module configured to receive signals from a touch screenwhich is overlaid on the display 228.

Also part of this exemplary mobile device is a speaker 234 andmicrophone 238. The speaker 234 and microphone 238 may be controlled bythe processor 208. The microphone 238 is configured to receive andconvert audio signals to electrical signals based on processor 208control. Likewise, the processor 208 may activate the speaker 234 togenerate audio signals. These devices operate as is understood in theart and as such are not described in detail herein.

Also connected to one or more of the buses 212 is a first wirelesstransceiver 240 and a second wireless transceiver 244, each of whichconnect to respective antennas 248, 252. The first and secondtransceiver 240, 244 are configured to receive incoming signals from aremote transmitter and perform analog front-end processing on thesignals to generate analog baseband signals. The incoming signal maybefurther processed by conversion to a digital format, such as by ananalog to digital converter, for subsequent processing by the processor208. Likewise, the first and second transceiver 240, 244 are configuredto receive outgoing signals from the processor 208, or another componentof the mobile device 208, and up convert these signal from baseband toRF frequency for transmission over the respective antenna 248, 252.Although shown with a first wireless transceiver 240 and a secondwireless transceiver 244, it is contemplated that the mobile device 200may have only one such system or two or more transceivers. For example,some devices are tri-band or quad-band capable, or have Bluetooth®, NFC,or other communication capability.

It is contemplated that the mobile device, and hence the first wirelesstransceiver 240 and a second wireless transceiver 244 may be configuredto operate according to any presently existing or future developedwireless standard including, but not limited to, Bluetooth, WI-FI suchas IEEE 802.11 a, b, g, n, wireless LAN, WMAN, broadband fixed access,WiMAX, any cellular technology including CDMA, GSM, EDGE, 3G, 4G, 5G,TDMA, AMPS, FRS, GMRS, citizen band radio, VHF, AM, FM, and wirelessUSB.

Also part of the mobile device is one or more systems connected to thesecond bus 212B which also interface with the processor 208. Thesedevices include a global positioning system (GPS) module 260 withassociated antenna 262. The GPS module 260 can receive and processsignals from satellites or other transponders to generate location dataregarding the location, direction of travel, and speed of the GPS module260. GPS is generally understood in the art and hence not described indetail herein. A gyroscope 264 connects to the bus 212B to generate andprovide orientation data regarding the orientation of the mobile device204. A magnetometer 268 is provided to provide directional informationto the mobile device 204. An accelerometer 272 connects to the bus 212Bto provide information or data regarding shocks or forces experienced bythe mobile device. In one configuration, the accelerometer 272 andgyroscope 264 generate and provide data to the processor 208 to indicatea movement path and orientation of the mobile device.

One or more cameras (still, video, or both) 276 are provided to captureimage data for storage in the memory 210 and/or for possibletransmission over a wireless or wired link or for viewing later. The oneor more cameras 276 may be configured to detect an image using visiblelight and/or near-infrared light. The cameras 276 may also be configuredto utilize image intensification, active illumination, or thermal visionto obtain images in dark environments. The processor 208 may processimage data to perform image recognition, such as in the case of, facialdetection, item detection, facial recognition, item recognition, orbar/box code reading.

A flasher and/or flashlight 280, such as an LED light, are provided andare processor controllable. The flasher or flashlight 280 may serve as astrobe or traditional flashlight. The flasher or flashlight 280 may alsobe configured to emit near-infrared light. A power management module 284interfaces with or monitors the battery 220 to manage power consumption,control battery charging, and provide supply voltages to the variousdevices which may require different power requirements.

FIG. 3 illustrates exemplary software modules that are part of themobile device and server. Other software modules may be provided toprovide the functionality described below. It is provided that for thefunctionality described herein there is matching software(non-transitory machine-readable code, machine executable instructionsor code) configured to execute the functionality. The software would bestored on a memory and executable by a processor.

In this example confirmation, the mobile device 304 includes a receivemodule 320 and a transmit module 322. These software modules areconfigured to receive and transmit data to remote device, such ascameras, glasses, servers, cellular towers, or WIFI system, such asrouter or access points.

Also part of the mobile device 304 is a location detection module 324configured to determine the location of the mobile device, such as withtriangulation or GPS. An account setting module 326 is provided toestablish, store, and allow a user to adjust account settings. A log inmodule 328 is also provided to allow a user to log in, such as withpassword protection, to the mobile device 304. A facial detection module308 is provided to execute facial detection algorithms while a facialrecognition module 321 includes software code that recognizes the faceor facial features of a user, such as to create numeric values whichrepresent one or more facial features (facial biometric information)that are unique to the user.

An information display module 314 controls the display of information tothe user of the mobile device. The display may occur on the screen ofthe mobile device or watch. A user input/output module 316 is configuredto accept data from and display data to the user. A local interface 318is configured to interface with other local devices, such as usingBluetooth® or other shorter-range communication, or wired links usingconnectors to connected cameras, batteries, data storage elements. Allof the software (with associated hardware) shown in the mobile device304 operate to provide the functionality described herein.

Also shown in FIG. 3 is the server software module 350. These modulesare located remotely from the mobile device and can be located on anyserver or remote processing element. As is understood in the art,networks and network data use a distributed processing approach withmultiple servers and databases operating together to provide a unifiedserver. As a result, it is contemplated that the module shown in theserver block 350 may not all be located at the same server or at thesame physical location.

As shown in FIG. 3, the server 350 includes a receive module 352 and atransmit module 354. These software modules are configured to receiveand transmit data to remote devices, such as cameras, watches, glasses,servers, cellular towers, or WIFI systems, such as router or accesspoints.

An information display module 356 controls a display of information atthe server 350. A user input/output module 358 controls a user interfacein connection with the local interface module 360. Also located on theserver side of the system is a facial recognition module 366 that isconfigured to process the image data from the mobile device. The facialrecognition module 366 may process the image data to generate facialdata (biometric information) and perform a compare function in relationto other facial data to determine a facial match as part of an identifydetermination.

A database interface 368 enables communication with one or moredatabases that contain information used by the server modules. Alocation detection module 370 may utilize the location data from themobile device 304 for processing and to increase accuracy. Likewise, anaccount settings module 372 controls user accounts and may interfacewith the account settings module 326 of the mobile device 304. Asecondary server interface 374 is provided to interface and communicatewith one or more other servers.

One or more databases or database interfaces are provided to facilitatecommunication with and searching of databases. In this exampleembodiment the system includes an image database that contains images orimage data for one or more people. This database interface 362 may beused to access image data users as part of the identity match process.Also part of this embodiment is a personal data database interface 376and privacy settings data module 364. These two modules 376, 364 operateto establish privacy setting for individuals and to access a databasethat may contain privacy settings.

Authentication System

An authentication system with path parameters that is operable in theabove described environment and system will now be described inconnection with FIG. 4. FIG. 4 shows a method for performing facialrecognition authentication with path parameters according to oneembodiment of the invention. As will be described in more detail below,the system utilizes the features of the mobile device 112 and server 120defined above to generate a secure and convenient login system as oneexample of an authentication system. This reduces the burden of the userhaving to type in complex passwords onto a small screen of a mobiledevice, prevents fraud through means such as key logging or screen shotcaptures, and increases security by combining several path parametersand/or device parameters which must be met before user is authenticated.

In step 410, the system enrolls a user in the facial recognitionauthentication system. In one embodiment, an authentication server, suchas the server 120 (FIG. 1), may be configured to authenticate a user toallow access to a user's account, such as a bank or other account, viathe mobile device 112. The authentication server 120 may be included asa part of a server of the institution or entity providing user accounts(hereinafter “account server”), or the authentication server may beprovided separately. For example, in the environment shown in FIG. 1,Servers 120B and 120C may represent account servers. In otherembodiments, the account server and the authentication server are one inthe same. In one embodiment, the authentication server 120 may providean authentication application to the user for installation on the mobiledevice 112.

An enrollment process according to one embodiment will be described withreference to FIG. 5. In this embodiment, a user via a mobile device 112establishes a connection between the mobile device 112 and the accountserver 120B in step 510. As just one example, the user may establish aconnection with a server of a financial institution such as a bank, orthis connection may occur later in the process after authentication. Theuser then provides typical login information to authenticate the user,such as a user name and password for a financial account in step 512. Instep 514, the user may next receive a prompt at the mobile device 112 toenroll in the facial recognition authentication system. The user then,via the user interface, indicates that he or she would like to set upthe authentication system in response to the prompt.

Next, in step 516, the mobile device 112 may send device information tothe authentication server 120. The device information may include amongother information a device identifier that uniquely identifies themobile device of the user. Such information may include devicemanufacturer, model number, serial number, and mobile networkinformation. In step 518, when the authentication server 120 isincorporated with the account server 120B, the authentication server 120associates and stores the device information with the user's accountinformation. When the authentication server 120 is separate from theaccount server 120B, the account server 120B may generate a uniqueidentifier related to the account information and send the uniqueidentifier to the authentication server 120. The authentication server120 may associate the device information and the unique identifier witheach other and may store the information in a database 124.

The user is next prompted to provide a plurality of images of his or herface using a camera 114 on the mobile device 112 (hereinafter,“enrollment images”) in step 510. The enrollment images of the user'sface are taken as the user holds the mobile device and moves the mobiledevice to different positions relative to his or her head and face.Thus, the enrollment images of the user's face are taken from manydifferent angles or positions. Furthermore, the path parameters of themobile device are monitored and recorded for future comparison in step522. Some non-limiting examples of how a user might hold a mobile deviceand take a plurality of images of her face is shown in FIGS. 6A-7B.

In FIGS. 6A and 6B, the user holds the mobile device 112 on one side ofhis or her face and moves the mobile device 112 in an arc like pathhorizontally about his or her face until the mobile device 112 is on theother side of her or her face. In FIGS. 7A and 7B, the user holds themobile device 112 far away from his or her face, and then brings themobile device 112 forward closer to his or her face. Of course, anynumber of other paths may be used in addition to those shown in FIGS.6A-7B. Additionally, the user may move his or her head while the camerais held fixed. The user could also hold the camera steady and move theirhead in relation to the camera. This method thus can be implemented witha webcam on a laptop or desktop, or on any other device, such as an IoTdevice where a camera is mounted on a similarly stationary location orobject.

The enrollment images may be obtained as follows. The user holds andorients a mobile device 112 with a camera 114 so that the camera 114 ispositioned to image the user's face. For example, the user may use afront facing camera 114 on a mobile device 112 with a display screen andmay confirm on the display screen that his or her face is in position tobe imaged by the camera 114.

Once the user has oriented the device, the device may begin obtainingthe enrollment images of the user. In one embodiment, the user may pressa button on the device 112 such as on a touchscreen or other button onthe device to initiate the obtaining of the enrollment images. The userthen moves the mobile device to different positions relative to his orher head as the device images the user's face from a plurality of anglesor positions as described above. When the above-mentioned front-facingcamera is used, the user may continually confirm that his or her face isbeing imaged by viewing the imaging on the display screen. The user mayagain press the button to indicate that the imaging is completed.Alternatively, the user may hold the button during imaging, and thenrelease the button to indicate that imaging is complete.

As described above, the mobile device 112 may include face detection. Inthis embodiment in step 524, the mobile device may detect the user'sface in each of the enrollment images, crop the images to include onlythe user's face, and send, via a network, the images to theauthentication server 120. In step 526, upon receipt of the enrollmentimages, the authentication server 120 performs facial recognition on theimages to determine biometric information (“enrollment biometrics”) forthe user. The authentication server 120 may then associate theenrollment biometrics with the device information and the uniqueidentifier (or account information) and stores the biometric informationin the database 124 in step 528. For added security, in step 530, themobile device 112 and the authentication server 120 may be configured todelete the enrollment images after the enrollment biometrics of the userare obtained.

In another embodiment, the mobile device 112 may send the images to theauthentication server 120 without performing face detection. Theauthentication server 120 may then perform the face detection, facialrecognition, and biometric information processing. In anotherembodiment, the mobile device 112 may be configured to perform thefacial detection, facial recognition, and biometric processing, and thensend the results or data resulting from the processing to theauthentication server 120 to be associated with the unique identifier oruser account. This prevents sensitive personal data (images) fromleaving the user's device. In yet another embodiment, the mobile device112 may perform each of the above-mentioned steps, and the mobile device112 may store the enrollment information without sending any of theenrollment biometrics or images to the server.

In one embodiment, the mobile device's gyroscope, magnetometer, andaccelerometer are configured to generate and store data while the usermoves the mobile device about his or her head to obtain the enrollmentimages (path parameters). The mobile device may process this data instep 532 to determine a path or arc in which the mobile device movedwhile the user imaged his or her face (“enrollment movement”). By usingdata from the accelerometer, magnetometer, and gyroscope, the system maycheck when a user is ready to begin scanning himself/herself, as well asdetermining the scan path. The data is thus used to determine when tostart and stop the scan interval. The data may additionally include thetime elapsed during scanning. This time may be measured from the userpressing the button to start and stop the imaging or may be measuredfrom the duration the button is held down while imaging, or during moremovement or to complete sweep.

The enrollment movement of the mobile device 112 (which is data thatdefined the movement of the mobile device during image capture) may besent to the authentication server 120. The authentication server 120associates and stores the enrollment movement, the enrollmentbiometrics, the device information, and the unique identifier or accountinformation. Alternatively, the data generated by the gyroscope,magnetometer, and accelerometer may be sent to the server 120, and theserver 120 may process the data to determine the enrollment movement. Inanother embodiment, the enrollment movement may be stored on the mobiledevice 112, such that all enrollment information is maintained on themobile device 112.

Thus, in the above described embodiment, the enrollment information maythus comprise the device information, the enrollment biometrics, and theenrollment movement (based on movement of the mobile device 112).

Returning to FIG. 4, once enrollment is complete, the authenticationserver 120 may later receive credentials from a user attempting toauthenticate with the system as shown in step 420. For example, a usermay attempt to log in to a user account. When a user attempts to log in,instead of or in addition to providing typical account credentials suchas user name and password, the user may again take a plurality of imagesor video of his or her face as the mobile device 112 is held in the handand moved to different positions relative to the head (“authenticationimages”) in the same manner as was done during enrollment (such as shownin FIGS. 6A-7B). In this manner, the user may provide the necessaryimages (the term images includes video as video is a succession ofimages) from many different angles and/or positions, and may providepath parameters of the device while obtaining the images(“authentication movement”) to both confirm the identity of the user aswell as the liveness and realness of that individual to ensure it is nota video, screen shot, or other representation of the person.

In one embodiment outlined in FIG. 8, the user via the mobile device 112obtains several authentication images in step 810 while moving themobile device 112 to different positions relative to the user's head.Using facial detection in step 812, the mobile device 112 detects theuser's face in each of the authentication images, crops the images, andsends the images to the authentication server 120. In anotherembodiment, the mobile device 112 sends the images to the server 124,and the server 124 performs facial detection. In step 814, theauthentication routing 120 may perform facial recognition on theauthentication images to obtain biometric information (“authenticationbiometrics”). In another embodiment, the mobile device 112 performsfacial recognition to obtain the authentication biometrics and sends theauthentication biometrics to the server 120.

In step 816, the mobile device 112 sends the device informationidentifying the device and sends path parameters such as gyroscope,magnetometer, and accelerometer information defining the path of themobile device taken during imaging, as well as the elapsed time duringimaging (“authentication movement”) to the server 120. The credentialsreceived by the authentication server 120 for a login in the facialrecognition system may thus comprise the device information, theauthentication images or the authentication biometrics, and theauthentication movement (path parameters).

Returning to FIG. 4, in step 430, the authentication server 120 verifiesthat the credentials received from the mobile device 112 sufficientlycorrespond with the information obtained during enrollment. For example,as shown in step 910 in FIG. 9, by using algorithms to process thecharacteristics of the face and light striking the face between thedifferent images, the authentication server 120 can determine that theface in the authentication images is three-dimensional, i.e. not arepresentation on a printed picture or video screen. Where the mobiledevice 120 sends only the authentication biometrics 120 to the server,the server 120 may validate the realness or three-dimensional aspects ofthe user imaged by comparing the biometric results of the differentimages.

In step 920, the authentication server 120 may then compare the logincredentials with the information stored from the enrollment process. Instep 920, the server 120 compares the identification of the deviceobtained during the login process to that stored during enrollment. Instep 930, the authentication biometrics may be compared with theenrollment biometrics to determine whether they sufficiently correspondwith the enrollment biometrics. In step 940, the authentication movementmay be compared with the enrollment movement to determine whether itsufficiently corresponds with the enrollment movement.

In some embodiments, a copy of the enrollment information may be storedon the mobile device 112, and the mobile device 112 may verify that thecredentials received on the mobile device 112 sufficiently correspondwith the enrollment information. This would allow a user to securedocuments, files, or applications on the mobile device 112 itself inaddition to securing a user's account hosted on a remote device, such asthe authentication server 120, even when a connection to theauthentication server 120 may be temporarily unavailable, such as when auser does not have access to the Internet. Further, this would allow theuser to secure access to the mobile device 112 itself. Or enrollmentinfo may be stored on server.

Accordingly, in step 950, if the authentication server 120 or mobiledevice 112 determines that the enrollment information sufficientlycorresponds with the credentials received, then the server or mobiledevice may verify that the identification of the user attempting logincorresponds the account holder. This avoids the cumbersome process ofthe user having to manually type in a complex password using the smallscreen of the mobile device. Many passwords now require capital,non-text letter, lower case, and numbers.

The level of correspondence required to determine that the enrollmentinformation sufficiently corresponds with the authentication informationin the login attempt may be set in advance. For example, the level ofcorrespondence may be a 99.9% match rate between the enrollmentbiometrics and the authentication biometrics and a 90% match ratebetween the enrollment movement and the authentication movement. Therequired level of correspondence may be static or elastic based on theestablished thresholds.

For example, the required level of correspondence may be based on GPSinformation from the mobile device 112. In one embodiment, theauthentication server 120 may require a 99.9% match rate as the level ofcorrespondence when the GPS information of the mobile device correspondswith the location of the user's home or other authorized location(s). Incontrast, if the GPS information shows the device is in a foreigncountry far from the user's home, the authentication server may requirea 99.99% match rate as the level of correspondence or may be deniedentirely. Hence, the required match between pre-stored authenticationdata (enrollment information) and presently received authentication data(authentication information) is elastic in that the required percentagematch between path parameters or images my change depending on variousfactors, such as time of day, location, frequency of login attempt,date, or any other factor.

The required level of correspondence may additionally depend on time.For instance, if a second authentication attempt is made shortly after afirst authentication attempt in a location far from the firstauthentication location based on GPS information from the mobile device112, the level of correspondence threshold may be set higher. Forexample, a user can not travel from Seattle to New York in 1 hour.Likewise, login attempts at midnight to three in the morning may be asign of fraud for some users based on patterns of the users' usage.

The level of correspondence between the enrollment information and theauthentication information may be the result of compounding the variousparameters of the enrollment information and the authenticationinformation. For example, when the button hold time in theauthentication information is within 5% of the button hold time of theenrollment information, the correspondence of the button hold time mayconstitute 20% of the overall match. Similarly, when the motion pathtrajectory of the authentication information is within 10% of theenrollment information, the motion path trajectory may constitute 20% ofthe overall match. Further parameter match rates such as the face sizeand facial recognition match in the authentication information ascompared to the enrollment information may constitute the remaining 10%and 50% of the overall level of correspondence. In this manner, thetotal overall level of correspondence may be adjusted (total of allparameters being more than 75%, for example), or the match rate ofindividual parameters may be adjusted. For example, on a secondattempted login, the threshold match rate of one parameter may beincreased, or the overall level of correspondence for all parameters maybe increased. The threshold match rates may also be adjusted based onthe account being authenticated or other different desired levels ofsecurity.

Returning to FIG. 4, in step 440, the authentication server 120 maygrant or deny access based on the verification in step 430. For example,if the authentication server 120 verifies that the credentials match theenrollment information, then the server 120 may authenticate the user toallow access to the user's account. In the instance where theauthentication server 120 is separate from the account server 120B (suchas a bank's server), the authentication server 120 may transmit theunique identifier to the account server along with an indication thatthe identity of the user associated with the unique identifier has beenverified. The account server 120B may then authorize the user's mobiledevice 112 to transmit and receive data from the account server 120B. Ofcourse, all this may occur at only the account server 120B or on themobile device 112 itself.

Alternatively, if the credentials provided by the user are not verified,the authentication server may transmit a message to display on thescreen of the mobile device 112 indicating that the login attemptfailed. The authentication server 120 may then allow the user to tryagain to log in via the facial recognition login system, or theauthentication server 120 may require the user to enter typical accountcredentials, such as a user name and password.

In one embodiment, the server 120 may allow three consecutive failedlogin attempts before requiring a user name and password. If in one ofthe attempts, the required level of correspondence is met, then the usermay be verified, and access may be granted. According to one embodiment,the authentication server 120 may retain the information from eachsuccessive authentication attempt and combine the data from the multipleauthentication attempts to achieve more accurate facial biometricinformation of the person attempting to authenticate. In addition, thelevel of correspondence may be increased at each successive attempt toauthenticate. In addition, by averaging the path data (authenticationmovement) and/or image data (authentication images/biometrics) fromseveral login attempts, the login data (enrollment information) isperfected and improved.

Accordingly, the above described authentication system allows forauthentication to a remote server 120 or on the mobile device 112itself. This may be accomplished as described above by the mobile device112 capturing the authentication credentials, and the authenticationserver 120 processing and analyzing the credentials compared to theenrollment information (cloud processing and analysis); the mobiledevice 112 capturing the authentication credentials and processing thecredentials, and the authentication server 120 analyzing the credentialscompared to the enrollment information (mobile device processing, cloudanalysis); or the mobile device 112 capturing the authenticationcredentials, and processing and analyzing the credentials compared tothe enrollment information (mobile device processing and analysis).

Advantages and Features of the Embodiments

The above described system provides several advantages. As oneadvantage, the facial recognition authentication system provides asecure login. For example, if during a login attempt the camera of themobile device imaged a digital screen displaying a person rotating theirhead while the phone was not moving, the accelerometer, magnetometer,and gyroscope data would not detect any motion. Thus, the enrollmentmovement and the authentication movement would not correspond, and thelogin attempt would be denied.

In addition, because a plurality of images are used as enrollment imagesand authentication images, histograms or other photo manipulationtechniques may be used to determine if a digital screen is present inplace of a human face in the images. For example, the system may checkfor light frequency, changes in the captured images, or banding in animage which would indicate an electronic display generated the image,backlighting, suspicious changes in lighting, or conduct other analyseson the images by comparing the images to determine that the actual liveuser is indeed alive, present, and requesting authorization to login.

As yet another advantage, as explained above, not only must theenrollment biometrics sufficiently correspond to the authenticationbiometrics, but also the enrollment movement must match theauthentication movement, and the device information must match theenrollment device information. For example, an application may bedownloaded to a mobile device that has a digital camera. The applicationmay be a login application or may be an application from a financialinstitution or other entity with which the user has an account. The usermay then login to the application using typical login credential such asa website user name and password. Further, the user may have a devicecode from logging in on another device or may use the camera to scan QRcode or other such code to pair the device to their user account.

The user then holds the mobile device to move the mobile phone todifferent positions relative to his or her head while keeping his or herface visible to the camera as it is moved. As the mobile device ismoved, the camera takes the enrollment images of the face. Duringimaging, the speed and angle of the current user's mobile devicemovement is measured using the accelerometer, magnetometer, andgyroscope to generate the enrollment movement. Further continuousimaging and detection of the face throughout the process has been shownto prevent fraud. This is because a fraud attempt cannot be made byrotating images in and out of the front of the camera.

For example, a user may start the movement from right to left or fromleft to right as shown in FIGS. 6A and 6B. The movement may also be in afront and back direction as shown in FIGS. 7A and 7B. Any other movementmay be utilized such as starting in the center, then going right, andthen going back to center. Vertical and diagonal movements may also beused to further compound the complexity of the enrollment movement. Whenthe user then later attempts login, the user must repeat the motionpattern in the authentication movement to match the enrollment movementin addition to the biometric data and device information matching. Thus,the security of the system is greatly enhanced.

The system therefore provides enhanced security for authenticating auser who has a mobile device. As explained above, the system may use atleast any one or more of the following in any number of combinations tosecurely authenticate the user: physical device verification, mobilenetwork verification, facial recognition including the size of the facein the image, a face detected in every frame during the movement,accelerometer information, gyroscope information, magnetometerinformation, pixels per square inch, color bits per pixel, type ofimage, user entered code or pattern, and GPS information.

As another advantage, the facial recognition login system provides aconvenient manner for a user to login to an account with a mobiledevice. For example, once enrolled, a user does not need to enter a username and password on the small mobile device each time the user wishesto access the account. Instead, the user simply needs to image himselfor herself while mimicking the enrollment movement with the mobiledevice. This is especially advantageous with smaller mobile devices suchas mobile phones, smart watches, and the like. It also saves time,eliminates lost or forgotten passwords, prevents others from spying onpasswords as they are entered, and reduces error caused by typing.

The system may be further configured to allow a user to securely log onto multiple devices, or to allow users to securely share devices. In oneembodiment, the enrollment information may be stored on anauthentication server (or on “the cloud”) and thus is not associatedonly with the user's original device. This allows the user to use anynumber of suitable devices to authenticate with the authenticationserver. In this manner, a user may use a friend's phone (third partydevice) or other device to access his or her information, such asaccount information, address book information, email or other messaging,etc. by performing the authentication operation on any device.

For example, the user may provide an email address, user name code, orsimilar identifier on the friend's phone such that the authenticationserver compares the login information with enrollment information forthe user's account. This would indicate to the authentication serverwhich authentication profile to use, but does not by itself allow accessto the user's data, accounts, or tasks. Upon logging out of a friend'sphone, access to the user's information on the friend's phone isterminated. The provides the benefit of allowing a user to securelyaccess account or other authentication accessible information or tasksusing any device without having to type the user's password into thethird-party device, where it could be logged or copied. In a sense, theuser is the password such that the user's facial features and storedfacial data is the stored password and by providing the user's face tothe camera, the user is providing the password.

Such a system may also be implemented without storing the user'senrollment information on the remote server, but instead maintainingenrollment information on the user's device. For example, when the userrequests to log in to third-party device, the authentication server maysend an authentication request to the user's device. The user may thenauthenticate the action using the above described system by providingauthentication images and authentication movement to the user's device.The user's device then authenticates the user based on the enrollmentinformation stored on the user's device. This data could be stored inthe application or other secure location and highly encrypted. Theuser's device provides verification of whether the authentication wassuccessful to the authentication server, whereupon the authenticationserver authenticates the user on the third-party device.

Through cloud-based authentication, a single user may also securelytransfer data between authenticated devices. In one embodiment, a usermay own a first device, such as a mobile phone, and is authenticated onthe first device via the authentication system. The user may thenacquire a new device, such as a new phone, tablet computer, or otherdevice. Using the cloud-based authentication system, the user mayauthenticate on the new device and transfer data from the first deviceto the new device. The transfer of data may be completed via theInternet, a local network connection, a Bluetooth connection, a wiredconnection, or a near field communication. The authentication processmay also be part of a security check to resent or restore a system afterthe phone is lost or stolen. Thus, the authentication system may be usedto activate or authenticate a new device, with the authentication usedto verify the user of the new device.

Similarly, the system may facilitate secure access to a single shareddevice by multiple people to control content or other features on thedevice. In many cases, passwords can be viewed, copied, guessed, orotherwise detected, particularly when a device is shared by severalusers. The users may be, for example, family members including parentsand children, coworkers, or other relationships, such as students. Theauthentication system may allow each of the family members to log inbased on his or her own unique enrollment information associated with auser account.

The device may restrict access to certain content or features for one ormore of the certain user's accounts, such as children's user accounts,while allowing access to content and features for others, such as theparents' accounts. By using the authentication system for the shareddevice, the users such as children are unable to utilize a password totry and gain access to the restricted content because the authenticationsystem requires the presence of the parent for authentication, asexplained above. Thus, device sharing among users with differentprivileges is further secured and enhanced. Likewise, in a classroomsetting, a single device may be securely shared between multiple peoplefor testing, research, and grade reporting.

ADAPTATIONS AND MODIFICATIONS

Numerous modifications may be made to the above system and methodwithout departing from the scope of the invention. For example, theimages may be processed by a facial recognition algorithm on the deviceand may also be converted to biometric data on the device which is thencompared to previously created biometric data for an authorized user.Alternatively, the images from a device may be sent through a wired orwireless network where the facial recognition algorithms running on aseparate server can process the images, create biometric data andcompare that data against previously stored data that assigned to thatdevice.

Multiple Profiles for a Single User

Further, the photo enrollment process may be done multiple times for auser to create multiple user profiles. For example, the user may enrollwith profiles with and without glasses on, with and without otherwearable devices, in different lighting conditions, wearing hats, withdifferent hair styles, with or without facial or ear jewelry, or makingdifferent and unique faces, such as eyes closed, winking or tongue outto establish another level of uniqueness to each user profile. Such‘faces’ made by the user would not be available on the user's SocialMedia Pages and hence not available for copying, manipulation, and useduring a fraud attempt. Each set of enrollment images, enrollmentbiometrics, or both may be saved along with separate enrollmentmovement. In one embodiment at least three images are captured as themobile device completes the path. It is contemplated that any number ofimages may be captured.

Linking Enrollment Information

It is also contemplated that the enrollment process may be linked to anemail address, phone number, or other identifier. For example, a usermay sign up with an email address, complete one or more enrollments asdescribed above, and confirm the enrollments via the same email address.The email address may then further enhance the security of the system.For example, if a user unsuccessfully attempts to login via theauthentication system a predetermined number of times, such as threetimes for example, then the authentication system locks the account andsends an email to the email address informing the user of theunsuccessful login attempts. The email might also include one or morepictures of the person who failed to login and GPS or other data fromthe login attempt. The user may then confirm whether this was a validlogin attempt and reset the system, or the user may report the loginattempt as fraudulent. If there is a reported fraudulent login, or ifthere are too many lockouts, the system may delete the accountassociated with the email address to protect the user's security. Thus,future fraudulent attempts could not be possible.

Feedback Meters

To further facilitate imaging, the mobile device may include variousfeedback meters such as a movement meter or accuracy meter as shown inFIG. 10. In one embodiment, the mobile device 1012 may display amovement meter 1024 that indicates the amount of movement the mobiledevice 1012 makes as the user moves the mobile device 1012 to differentpositions relative to his/her head. For example, the movement meter 1024may be represented as a line that slides from one side of the screen. Inthis manner, the enrollment process may require a certain threshold ofdevice movement to register a user with the multi-dimensionalauthentication system. For example, the system could require that themobile device 1012 is moved in an arc or straight line and rotate atleast 45 degrees to create the enrollment information. In anotherexample, the system could require an acceleration experienced by thedevice exceeding a threshold amount. The movement meter may also aid theuser in learning how to image himself/herself using the authenticationsystem.

The mobile device 1012 may also display an accuracy meter 1026 or anyother visual representation of authenticated frames to aid the user inauthenticating himself/herself using the authentication system andlearning to improve authentication. The accuracy meter 1026 may show auser a match rate (graphical, alpha, or numerical) of a predeterminednumber of images obtained during the authentication process. Theaccuracy meter can be represented on the display in a variety of waysincluding numeric percentages, color representation, graphical, and thelike. A combination of representations may also be utilized.

For example, as shown in FIG. 10, match rates for a predetermined numberof images taken during authentication are represented on the accuracymeter. In the embodiment shown in FIG. 10, each of the images may berepresented by a column in a graph, and the accuracy can be shown foreach image in each column. For example, the column with a longer barrepresent higher accuracy, and a column with a lower bar representslower accuracy. In addition to match rates for images, the match ratesfor the path parameter may also be displayed. Over time the user canimprove.

In another embodiment, each of the images may be represented on a tableas a color that corresponds to the match rate. The color dark green mayrepresent a very high match rate, light green may represent a good matchrate, yellow may represent a satisfactory match rate, red may representa mediocre match rate, and grey may represent a poor match rate. Othercolors schemes may also be used.

The height of the bars or the colors used may correspond topredetermined match rates. For example, a full bar or dark green may bea match rate greater than 99.9%, a three-quarter bar or light green maybe a match rate between 90% and 99.9%, a half bar or yellow may be amatch rate of 50-90%, red may be a match rate of 20%-50%, and a singleline to a quarter bar or grey may be a match rate of 0-20%. A pie chart,line graph, or any other type of representation could also be used orany other numerical or graphical display. An overall score may bepresented or a score per image.

The accuracy meter may also include a message 1028 indicating an overallmatch score. For example, the accuracy meter may indicate an averageoverall match score or the number of images which achieved a 99.9% matchrate and display the message to a user. With the movement meter 1024 andthe accuracy meter 1026 as described above, the user may quickly learnto use the authentication system due to the feedback presented by themeters 1024, 1026.

Gamification and Rewards

The movement and accuracy meters 1024, 1026 may also be configured toincorporates game features, aspects, or techniques into theauthentication system to encourage a user to try and get the best matchpossible (such as a high number score or a high percentage of frames),increasing the user's skill in utilizing the authentication system. Thisalso builds user adoption rates for the technology.

For example, the user may compete with themselves to mimic or improvepast authentication scores to encourage or train the user to achieve ahigh score. Further modifications of the authentication meter may alsobe incorporated such as the ability to share accuracy match results withothers to demonstrate one's skill in using the system or to competeagainst others. In other instances, the user may receive a reward, suchas a gift or coupon, for high accuracy scores. While this may slightlyincrease costs, the reduction in fraud loss would far outweigh theadditional cost.

Further game techniques may be incorporated into the authenticationsystem to encourage users to take actions which will preventunauthorized or fraudulent authentication. In one embodiment, theauthentication system may award users that engage in fraud preventingactivities. One such activity is utilizing the facial recognitionauthentication system described herein. For example, based on the abovedescribed accuracy meter, the system may reward a user that successfullyauthenticates with the system above a certain match rate. The system mayaward reward points, cash, or other prizes based on the successfulauthentication or on a predetermined number of successfulauthentications. Where reward points are utilized, the points may becashed in for predetermined prizes.

Other game features may involve award levels for users who gain apredetermined amount of experience using the authentication feature. Forexample, different reward levels may be based on users successfullyauthenticating 100 times, 500 times, 1000 times, etc. Because eachinstance of fraud loss can be significant and can damage the goodwill ofthe business or organization, the benefits to fraud prevention aresignificant.

In one embodiment, the user may be notified that he or she has achievedvarious competency levels, such as a “silver level” upon achieving 100successful authentications, a “gold level” for achieving 500 successfulauthentications, or a “platinum level” for achieving 1000 successfulauthentications. A set number of points awarded for each authenticationabove a given match rate may increase based on the user's experiencelevel. Of course, the names of the levels and the number ofauthentications for each level as described above are only exemplary andmay vary as desired.

In one embodiment, an authentication only counts toward reward levelswhen business is transacted at the web site while in other embodiments,repeated attempts may be made, all of which count toward rewards.Another feature may incorporate a leaderboard where a user may benotified of a user ranking comparing his or her proficiency orwillingness in using the authentication system as compared with otherusers.

Successful use of the authentication system benefits companies andorganizations that utilize the system by reducing costs for fraudulentactivities and the costs of preventing fraudulent activities. Those costsavings may be utilized to fund the above described game features of theauthentication system.

Further activities that correspond to the authentication system andcontribute to the reduction of fraud may also be incorporated to allow auser to earn points or receive prizes. Such activities may include auser creating a sufficiently long and strong password that uses acertain number and combination of characters. This encourages andrewards users to set passwords that are not easily compromised. Otherexamples may include rewarding users to take time to performverification steps in addition to an initial authentication such as amobile phone or email verification of the authentication, answering oneor more personal questions, or other secondary verifications ascurrently known or later developed. This rewards users for taking onadded time and inconvenience to lower the risk of fraud to a company ororganization.

As another example, if the authentication service is used to login towebsites or apps that provide affiliate programs, then the reward orgift can be subsidized from the affiliate commissions on purchases madeon those sites. For example, if a commerce (product or service) web siteutilizes the method and apparatus disclosed herein to avoid fraud, andthus increase profits, then a percentage of each purchase made by a userusing the authentication service will be provided to the authenticationservice. By reducing fraud, consumer purchases are more likely andadditional users will be willing to enter financial and personalinformation. An affiliate link, code, or referral source or identifiermay be used to credit the authentication system with directing theconsumer to the commerce (product or service) web site.

Multiple Account Login

It is also contemplated that the authentication system may be configuredto allow a user to access many different web sites with a singleauthentication. Because the authentication process and result are uniqueto the user, the user may first designate which participating web sitesthe user elects to log into and then after selecting which one or moreweb sites to log into, the user performs the authentication describedherein. If the secure authentication is successful, then the user islogged into the selected web sites. In this way, the authenticationprocess is a universal access control for multiple different web sitesand prevents the user from having to remember multiple different usernames and passwords while also reducing fraud and password overhead foreach user.

Automatic Start/Stop of Imaging

It is also contemplated that the system may be configured to have thevideo camera running on the phone. The mobile device would capture imageframes and path parameter data when the phone moves (using the camera,gyroscope, magnetometer, and accelerometer) but only process intobiometric data on the device or send the frames up to the server if aface is detected in them. In other embodiment, no image data or framesis sent to the server and instead the analysis is performed on theuser's mobile device, such as a phone. In this embodiment, theapplication executing on the mobile device could trigger the softwareapplication to start saving frames once the phone is moving and then ifthe phone continues to move in the correct path (a semi-circle, forexample) and the system detects a face in the frame the mobile devicewould start to send images, a portion of the image, or biometric data tothe server for processing. When the system senses motion it may triggerthe capture of images at certain intervals. The application may thenprocess the frames to determine if the images contain a face. If theimages do include a face, then the application crops it out and thenverifies if the motion path of the mobile device is similar to the oneuse used during enrollment. If the motion path is sufficiently similar,then the application can send the frames one at a time to the server tobe scanned or processed as described above.

Banding and Edge Detection

When a fraudulent attempt is made using a display screen, such as anLED, LCD, or other screen, the system may detect the fraudulent loginattempt based on expected attributes of the screen. In one embodiment,the authentication system will run checks for banding produced bydigital screens. When banding is detected, the system may recognize afraudulent attempt at a login. In another embodiment, the system willrun checks for edge detection of digital screens. As the mobile deviceis moved to obtain the authentication movement during a login attempt,the system checks the captured images to for edges of a screen torecognize a fraudulent login attempt. The system may also check forother image artifacts resulting from a screen such as glare detection.Any now know or later developed algorithms for banding and screen edgedetection may be utilized. Upon detection of fraud will preventauthentication and access to the website or prevent the transaction oraccount access.

Other Attributes Estimation

The authentication system may further conduct an analysis on theenrollment images to estimate at least one of a gender, an approximateage, and an ethnicity. In an alternative embodiment, the user maymanually enter one or more of their gender, an approximate age, and anethnicity, or this information may be taken or obtained from existingrecords which are known to be accurate. The authentication system maythen further store a user's estimated gender, age, and ethnicity asenrollment credentials or user data. Thus, when the user later attemptsto authenticate with the system, the system will compare derived gender,age, and ethnicity obtained from authentication images (using biometricanalysis to determine such data or estimates thereof based onprocessing) with the stored gender, age, and ethnicity to determinewhether to authenticate the user. For example, if the derived data forgender, age and ethnicity matches the stored enrollment credentials,then the authentication is successful, or this aspect of theauthentication is successful.

The authentication system may make the gender, age, and ethnicityestimations based on a single image during the authentication process orbased on multiple images. For example, the authentication system may usean image from the plurality of images that has an optimal viewing angleof the user's face for the analysis. In other embodiments, a differentimage may be used for each analysis of age, gender, and ethnicity whendifferent images reveal the best data for the analysis. Theauthentication may also estimate the gender, age, and ethnicity in aplurality of the images and average the results to obtain overall scoresfor a gender, age, and ethnicity.

As an alternative to obtaining the gender, age, and ethnicity asenrollment information, the estimated gender, age, and ethnicityestimations as authentication credentials may be set over a course ofrepeated use of the authentication system. For example, if in previoussuccessful authentications using biometrics and movement information,the authentication system always estimates a user's age being between 40and 50, then the authentication may set credentials for that userrequiring later login information to include images of a face estimatedto be between 40 and 50. Alternatively, gender, age, and ethnicityestimations may be implemented as one of many factors contributing to anoverall authentication score to determine whether or not to authenticatea user.

For example, if the authentication process has a gender estimation of +or −0.2 of 1.9 male rating, then if the actual results do not fallwithin that range the system may deny access for the user. Likewise, ifthe user's age range always falls between 40-50 years of age duringprior authentication attempts or enrollment, and an authenticationattempt falls outside that range, the system may deny access or use theresult as a compounding factor to deny access.

In a further embodiment, when a bracelet or watch capable of obtainingan EKG signature is used, a certain EKG signature may be required atlogin. The EKG signature could also be paired with the facialrecognition rotation to provide multiple stage sign-on for criticalsecurity and identification applications. Further, the credentials couldalso include GPS information where login is only allowed within certaingeographic locations as defined during enrollment. In one configurationthe GPS coordinates of the mobile device are recorded and logged for alogin attempt or actual login. This is additional information regardingthe location of the user. For example, if the GPS coordinates are in aforeign country known for fraud, then the attempt was likely fraudulent,but if the GPS coordinate indicate the attempt or login was made in theuser's house, then fraud is less likely. In addition, some applicationsmay only allow a user to login when at specified location such as asecure government facility or at a hospital.

The enrollment information may further include distance information.Because the motion arc (speed, angle, duration . . . ) is unique to eachuser, face detection software on the device can process the images anddetermine if the device is too close or too far from the subject. Or inother words, the enrollment information may consider the size of theface in the images. Thus, the potential enrollment information may alsovary based on the length of a user's arm, head, and face size, and onthe optics of the camera in the user's particular mobile device. Theuser may also be positioned at a fixed computer or camera, such aslaptop, desktop, or atm. The user may then move the face either forwardsand back, side to side, or up and down (or a combination) to create theimages. Hence, this method of operation is not limited to a mobiledevice. In one embodiment, the camera is located in an automobile, suchas in a mirror, and the person moves their head or face to authenticate.

Gradual Authentication Access

In one embodiment, the system is set to limit what the user can do whenfirst enrolled and authenticated. Then, after further authentications orafter a predetermined time period and number of authentications,additional capabilities may be granted. For example, during the first 20authentications during the first 3 months, a maximum transaction of $100may be allowed. This builds a database of known authentication data inconnection with non-objected to transactions by the user. Then, duringthe next 20 authentications a transaction limit of $3000 may beestablished. This limits the total loss in the event of fraud when theauthentication data is limited, and the user is new to the system. Forexample, if an unauthorized user is able to fraudulently enroll in theauthentication system.

Video Display for Imaging

When the user images himself/herself using a front-facing camera, theuser may confirm that his/her face is being imaged by viewing the imageon the display, as described above. The image shown on the display maybe configured to be smaller in area than the entire display and may bepositioned in an upper portion of the display towards the top of thedevice. When the user's image is shown only in the top portion of theuser's display screen, the user's eyes tend to look more closely at thefront camera. When the user's eyes are tracking up, the accuracy of thefacial recognition may be improved. Further, tracking the movement ofthe eyes from frame to frame may allow the system to validate that theimages are of a live person, and are not from a photograph or videorecording of the person.

The image shown on the display may also be positioned to correspond witha camera location on the user's device, as shown in FIGS. 11A-11C.Mobile devices that are available today may include front-facing camerasdisposed at many different positions. For example, one mobile device1112 a, 1112 b may have a front-facing camera 1114 a, 1114 b that isdisposed above the display and off center towards one side or the other,as shown in FIGS. 11A and 11B. Accordingly, the feedback image 1116 a,1116 b of the user shown on the display may be positioned to correspondwith the location of the camera 1114 a, 1114 b as shown. In FIG. 11A,where a camera 1114 a is above the display and is off-center at aposition left of the center, then the image 1116 a may be shown in anupper left corner of the display. In FIG. 11B, where a camera 1114 b isabove the display and is off-center at a position right of the center,then the image 1116 b may be shown in an upper right corner of thedisplay. As shown in FIG. 11C, a mobile device 1112 c may have a camera1114 c that is disposed centered directly above the display. There, theimage 1116 c may be displayed centered in an upper portion of thedisplay. In this manner, a user's eyes are directed close to and/ortrack as close to the camera as possible, aiding eye tracking andmovement verification. The user is also able to better see the feedbackimage, and other feedback or information on the screen, as they move themobile device.

The image viewed on the display by the user may further be modified suchthat the edge pixels on the sides display are stretched horizontally asshown in FIG. 12. That is, a predetermined area 1206, 1208 on both theright and the left sides are warped to stretch towards right and leftedges, respectively, of the screen. This allows a larger verticalportion of the displayed image to be shown on the display.Simultaneously, this trains a user to use the system correctly bykeeping his or her face in the center of the screen, as his or her facewould become warped on the screen if it becomes off center and part ofthe face enters the one of the warped areas.

Authentication in Low-Light Environments

To facilitate imaging, the screen on the mobile device may additionallybe displayed with a white background, and the brightness of the screenmay be increased to light up the user's face in dark environment. Forexample, a portion of the display could provide video feedback for theuser to ensure he or she is imaging himself or herself, while theremaining portion of the display is configured to display a bright whitecolor. Referring to the example shown in FIG. 11C, this may be done byshowing the video feedback 1116 c on a center of the display, with thesurrounding areas being displayed as bright white bars around the videofeedback 1116 c. In very dark situation, an LED flash on the back sideof the mobile device and the back facing camera may be used.Alternatively, the camera may be configured to create an image usinginfrared light or other night vision techniques.

When infrared imaging is used as thermal imaging, further securityenhancements are possible. Particularly, the thermal imaging may beanalyzed to indicate whether the obtained images are from an actual useror are fraudulent images from a screen or other device. When a person isin front of an infrared thermal imaging camera, the heat radiationdetected should be fairly oval shaped designating the person's head. Incontrast, the heat radiating from a screen is typically rectangular.Further, the heat patterns detected in the actual person's face as wellas the movement of the heat patterns in the images can be compared withexpected heat patterns of a human face to distinguish the images fromfraudulent authorization attempts using a screen.

Detecting Output from the Mobile Device

The display or other light source on the mobile device may further beutilized to provide additional security measures. During theauthentication process described above, light from the display or otherlight source is projected onto the user's face and eyes. This projectedlight may then be detected by the camera of the mobile device duringimaging. For example, the color tone detected on the skin, or areflection of the light off the cornea of a user's eye may be imaged bythe camera on the mobile phone. Because of this, random light patterns,colors, and designs may be utilized to offer further security and ensurethere is a live person attempting authentication and not merely an imageor video of a person being imaged by a fraudster.

As one example, when a user begins authentication, the authenticationserver may generate and send instructions to the user's device todisplay a random sequence of colors at random intervals. Theauthentication server stores the randomly generated sequence for latercomparison with the authentication information received from the mobiledevice. During authentication imaging, the colors displayed by thedevice are projected onto the user's face and are reflected off theuser's eyes (the cornea of the eyes) or any other surface that receivesand reflects the light from the screen. The camera on the user's mobiledevice detects the colors that are reflected off the user's skin or eyes(or other surface) and generates color data indicating the colorsdetected based on the screen projection. This data may be returned tothe authentication server to determine if the color sequence or patternsent to the mobile device matches that known sequence or patternprojected by the screen of the user device. Based on this comparison atthe authentication server the authentication is a success or denied. Thecomparison with the random sequence of colors in the instructions mayalternatively occur exclusively at the user device to determine that alive user is being authenticated.

As another example, when a user begins authentication, theauthentication server may send instructions the user's device to displaya randomly generated pattern which is then stored on the authenticationserver. This pattern may include graphics, text, lines or bars, flashinglight patters, colors, a QR code, or the like. The randomly generatedpattern is displayed during authentication imaging, and the pattern isreflected off the user's eyes (cornea). The camera of the user's devicedetects the reflected pattern off the eye of the user and processes thereflected, mirrored image of the displayed pattern. The processedpattern (such as being converted to a numeric value) is transmitted tothe authentication server and compared to the pattern that was randomlygenerated and stored on the authentication server to verify if thepattern displayed by the screen and imaged after reflection off theuser's face establishes a pattern match.

If a match occurs, this establishes or increases the likelihood that alive person is being imaged by the device. If the pattern is not amatch, or does not meet a match threshold level, then the authenticationprocess may fail (access denied) or the account access or transactionamount may be limited. It is noted that this example could also beincorporated on desktop computer with a webcam that does not incorporatethe enrollment movement and authentication movement described above.Further, this example may not only be incorporated with facialrecognition, but could also serve as an added layer of security for irisrecognition or any other type of eye blood vessel recognition, or anyfacial feature that is unique to a user.

When the above example is implemented on a desktop computer, eyetracking may also be utilized to further demonstrate the presence of alive user. For example, the screen could show a ball or other randomobject or symbol moving in a random pattern that the user watches withhis or her eyes. The camera can detect this real time movement to verifythe user is live, and not a picture or display, and verify that the eyeor head movements correspond to and match the expected movement of theobject or words on the screen, which are known by the authenticationsystem. Eye tracking can also be done by establishing an anchor point,such as via a mouse click at a location on the screen (assuming that theuser is looking at the location where the mouse click takes place), andthen estimating where the user is looking at the screen relative to theanchor position.

The use of a moving object on the screen may also be beneficial duringenrollment on either a mobile or stationary device. For example, whilecapturing the enrollment images, the device may display a moving digitalobject (such as a circle or words(s)) that moves around the screen sothat the user is encouraged to follow it with his or her head and eyes.This movement may be involuntary from the user, or the device may beconfigured to instruct the user to follow the object. This results inmovement of the head and/or eyes creating small changes in theorientation of the user's head and face with the device camera,providing more complete enrollment information. With more completeenrollment information, the system may better ensure that the user willlater be authenticated at a high rate even at slightly different anglesduring future authentication attempts.

Intuitive User Training and Enhanced Security by “Zooming”

In one embodiment, the system is configured to aid the user to easilylearn to authenticate with the system. As shown in FIG. 13A, onceenrollment or authentication is begun as described previously, thesystem causes the user's mobile device 1310 to display a small oval 1320on the screen 1315 while the mobile device 1310 is imaging the user.Instructions 1325 displayed on the screen 1315 instruct the user to holdthe mobile device 1310 so that his or her face or head appears within inthe oval 1320. Because the oval 1320 is small, the user is required tohold the mobile device 1310 away from his or her body, such as bystraightening his or her arm while holding the mobile device 1310. Themaximum arm length and face size is unique to the user. In otherembodiment, the arm may not be fully straightened such as to accommodateoperation when space is not available, such as in a car or in a crowdedlocation. It is noted that while the small oval 1320 is shown centeredin the display, it may be positioned anywhere on the screen 1315.

Next, as shown in FIG. 13B, the system causes the user's mobile device1310 to display a larger oval 1330 on the display 1315. The display 1315may also show corresponding instructions 1335 directing the user to“zoom in” on his or her face to fill the oval 1330 with his or her face.The user does this by bringing the mobile device 1310 closer to his orher face in a generally straight line to the user's face (such as shownin FIGS. 7A and 7B) until the user's face fills the oval 1330 or exceedsthe oval. In other embodiments, the large oval 1330 may simply be aprompt for the user to bring the mobile device 1310 closer to the user'sface.

Thus, the system provides and teaches the user a simple method toprovide enrollment and authentication images along with enrollment andauthentication movement as explained above. The system may also teachvarying enrollment and authentication movement by varying the locationof the small oval 1320 on the screen 1315, and by changing the order andthe size of the ovals displayed. For example, the user may zoom in halfway, then out, then in all the way, by moving the mobile device. Thesystem may be configured to monitor that the camera's zoom function(when equipped) is not in use, which typically requires the user totouch the screen.

In one embodiment, the enrollment movement may be omitted, and theauthentication movement may be compared to expected movement based onthe prompts on the screen. For example, the device or authenticationserver generates a series of differently sized ovals within which theuser must place his or her face by moving the mobile device held in theuser's hand. In this manner, the authentication movement may bedifferent during each login depending on the order, size, and placementof the ovals shown on the screen.

The system may also incorporate other security features when the “zoomin” movement is used as shown in FIGS. 13A and 13B. The zoom in featureis typically from moving the device closer or further from the user, buta camera zoom function is also contemplated. When images are taken atrelatively close distances between the subject and the camera, theimages are distorted through what is known as perspective distortion.Some texts may refer to this distortion as fish-eye type distortion, butperspective distortion is a more accurate technical terminology.Further, lenses configured as wide-angle lenses may contribute barreldistortion (a type of lens distortion) in an image. Other types of lensdistortions are also present. These distortions may be tracked fordifferent types of lenses for different devices. The degree ofdistortion experienced by a user performing the “zoom in” movement tofit their face within the screen displayed prompts may vary and isdependent on the type of optics used in the camera's lens, the type ofdistortion, and other factors.

The distortion becomes more obvious on an image of a person's face whenthe person images his or her face close to the lens. The effect resultsin the relative dimensions of the person's face appearing different thanwhen the imaging is done with the person's face farther away from thelens. For example, a person's nose may appear as much wider and tallerrelative to a person's face when the image is taken at a close proximityas compared to when the image is taken at a distance. The differences inthe relative dimensions are caused by the relatively larger differencesin distances between the various facial features and the camera when theperson is imaged close to the lens as compared to the relatively equaldistances between the facial features and the camera when the person isimaged at a distance farther from the lens.

Such differences have been found to be significant in many facialrecognition algorithms. That is, a facial recognition algorithm may notresult in a high likelihood of a match between images of a live personimaged at a close proximity and the same person imaged at a farproximity. In contrast, if a two-dimensional photograph of a person isimaged by the camera at both a close proximity and a farther proximity,the relative distances (length) between the lens and the facial featuresof the two-dimensional image do not change so significantly. Thus, afacial recognition algorithm would recognize the two-dimensionalphotograph as a high likelihood of a match when imaged at both a closeproximity and a distance farther from the lens.

This effect also changes what parts of the user may be viewed by thecamera of the user's mobile device based on the angle between the cameraand facial features of the user. For example, when imaged farther away,the camera may image the user's face with the user's ears visible in theimage. However, when the user device is moved close to the user's face,the image captured by the camera no longer includes the user's ears.This is because other facial features of the user's face, such as theuser's cheeks/cheekbones now block the line of sight from the camera tothe user's ears. In contrast, when a two-dimensional picture is imagedby the camera, the same facial features of the two-dimensional picturewill always be visible to the camera whether the camera is close to thetwo-dimensional picture or far away. Thus, the three-dimensions orliveness of the user's face may further be verified by detecting changesin what part of the user face or other features are present in theimage.

This effect as described above may be used to increase the security ofthe authentication system. For example, during enrollment, enrollmentimages may be provided by the user at both close and far proximitiesfrom the lens, in addition to other positions through the movement.Later, during authentication, authentication images may be obtained atboth the close and far distances from the lens to determine if theymatch with the enrollment information obtained from the enrollmentimages. Further, because perspective distortion is expected when anactual, three-dimensional person is present, an absence of the relativechange in the dimensions of the facial features alerts the system to afraudulent attempt at authentication. This effect could not easily bere-created with a two-dimensional picture (printed photograph or screen)and thus, this step can serve as a secure test to prevent atwo-dimensional picture (in place of a 3D face) from being used forauthentication.

In other words, using this movement of “zooming” in and out on theuser's face, two or more biometric profiles could be created for thesame person. One of the multiple profiles for the person may be imagedfarther from the camera, and one of the multiple profiles may be for theperson imaged closer to the camera. For the system to authenticate theperson, the authentication images and biometrics must match the two ormore profiles in the enrollment images and biometrics.

In addition, the system may detect the presence of a real person ascompared with a fraudulent photograph of a person by comparing thebackground of the images obtained at a close and a far proximity. Whenthe mobile device 1310 is held such that the person's face fits withinthe oval 1320, objects in the background that are almost directly behindthe person may be visible. However, when the mobile device 1310 is heldsuch that the person's face fits within the larger oval 1330, theperson's face blocks the cameras ability to see the same objects thatare almost directly behind the person. Thus, the system may compare thebackgrounds of the images obtained at the close and the far proximity todetermine whether the real person is attempting authentication with thesystem.

Of course, in FIGS. 13A and 13B, shapes or guides other than ovals 1320and 1330 may be used to guide the user to hold the mobile device 1310 atthe appropriate distance from his or her face. For example, the mobiledevice 1310 may show a full or partial square or rectangle frame.Further, the system may vary the size and location of the frame, such asthe ovals 1320, 1330 to add further security. For example, the systemmay require a medium sized frame, a small frame, and then a large frame.As another example, the system may require a small frame at a firstlocation and a second location, and then a large frame. This may be donerandomly to teach different users different enrollment andauthentication movements, or to increase the security of theauthentication system.

The number of frame sizes presented to the user may also vary for asingle user based on the results of other security features describedherein. For example, if the GPS coordinates of the mobile device showthat the device is in an unexpected location, more frames at differentdistances may be required for authentication. One or more indicators,such as lights, words, or symbols may be presented on the screen to bevisible to the user to direct the user to the desired distance that themobile device should be from the user.

In FIGS. 13A and 13B, the system may predict the expected perspectivedistortion of the images based on the mobile device used for enrollmentand authentication, and based on known and trusted enrollment data. Inaddition, or as an alternative, the known specifications of a mobilephone camera for a given model may be utilized to predict the expecteddistortion of the person's facial features at different distances fromthe lens. Thus, the authentication may be device dependent. Further,enrollment information from the user is not required at every possibledistance from the camera.

For example, as described above, enrollment images and biometrics may beobtained at at least two distances from the user. During authentication,multiple intermediary images are captured in addition to imagescorresponding the close and far distances of the enrollment images andbiometrics. Based on the expected pattern of distortion of theseintermediary images according to the distanced traveled by the device,the system may validate that the change in distortion of the images ishappening at the correct rate, even though only two enrollment profilesare obtained.

The capturing of these images may be still images or video, such thatframes or images are extracted from the video that is taken during themovement from the first position distant from the user and the secondposition proximate the user. Thus, it is contemplated the operation maycapture numerous frames during the zoom motion and ensure that thedistortion is happening at the correct rate for the head size and themovement of the mobile device distance based on data from theaccelerometers, magnetometers, and so forth.

Over time based on accumulated data, or calculated data during designphase, the system will have data indicating that if a phone is moved acertain distance toward a user's face, then the distortion effect shouldfall within a known percentage of the final distortion level or initialdistortion level. Thus, to fool or deceive the authentication systemdisclosed herein, the fraud attempt would not only need to distort thefraudulent two-dimensional picture image, but would also need to cut thebackground, and then make a video of the face, distortion, andbackground that does all of this incrementally and at the correct speed,all while not having any banding from the video screen or having anyscreen edges visible, which is very unlikely.

Many currently known facial detection and facial recognition algorithmsare configured to only look for a face within an image where thedepicted head is smaller than the image's dimensions. If a user's theforehead, ears or chin were not visible in the frame the remainder ofthe face would not be detected. Thus, to ensure that the facialdetection and recognition algorithms detect and recognize the user'sface in the zoomed in image (FIG. 13B), the system may add a largebuffer zone around the image taken at a close proximity. This creates alarger overall image and allows current facial detection and recognitionalgorithms to detect and recognize the face, even where the face of theuser is large in the original image. In one embodiment, the facedetection may be configured to detect portions of the face such as eyes,nose and mouth and extrapolate that the entire face is present based onone of more of those features being present.

When the enrollment and authentication movements resulting from theprocess described with FIGS. 13A and 13B is used, the eye trackingsecurity features described above may also be enhanced. For example,when the user is instructed to bring the mobile device 1310 closer tohis or her face to fill the oval 1330, the QR code, a random shape, abar code, color, text, numbers or any other visual indictor may bedisplayed on the screen. At this close distance, the reflection of thedisplayed indicator off the user's eye or face may be more easily imagedby the camera. Furthermore, eye movement, blinking, and the like todetermine the “liveness” of the person being imaged may also be moreeasily obtained at the close proximity.

In one embodiment, at least one blink is required to prove liveness forauthentication. In another embodiment, blinks may be counted, and thenumber of blinks may be averaged over time during authentications. Thisallows for an additional factor in authentication to be the number ofblinks observed during the motion. If a pattern of when the user blinksduring the motion is observed, the system may verify that the userblinks at the expected time and device location during the motion duringfuture authentication attempts. In some instances, the system may promptthe user to blink, wink, smile, etc. and monitor the captured images toverify that the user has performed the prompted action within apredetermined time. A series of prompted actions may be given to theuser to perform (for example, blink, wink right eye, then smile). In oneexample, the system may prevent authentication (lock out a user) if theprompted actions are performed out of order or are not performed withinthe time period because this may be indicative of a fraudulent recordingbeing used in place of a live user.

In other embodiments, the size or location of the oval or frame maychange to sizes or locations other than that shown in FIGS. 13A, 13Bsuch that the user must position and/or angle the phone to place his orher face within the oval. This establishes yet another method ofinsuring liveness of the user. The oval may start small and becomelarger or start large and become smaller. The shape may be shapes otherthan oval, such as square, triangular, rectangular, or any other shape.Instead of or in additional to a shape, text may be shown instructingthe user to move the phone closer or farther from the user's face.

In one exemplary method, the mobile device is positioned at a firstdistance from the user and a first image captured for processing. Thisdistance may be linearly away from the user and in this embodiment notin an arc or orbit. This may occur by the user moving the mobile device,either by hand, or by the mobile device being on a movable device orrail system. Or, the lens system may be adjusted if in a fixed system tochange the size of the user's face in relation to the frame size.Alternatively, the user may stay stationary, the multiple cameras may beused, or camera may move without the user moving. Once some form ofmovement (from a device, camera, lens, or user) has occurred toestablish the camera at a second distance, a second image is capturedfor processing. Movement from the first position to the second positionmay be straight toward the user. Processing occurs on both images.

The processing may include calculations to verify a difference betweenthe two images, or a difference in biometrics obtained from the twoimages, that indicates that a real person is being imaged. Processingmay occur to compare the first authentication image to a firstenrollment image (corresponding to the first distance) to determine if amatch is present and then compare the second authentication image to asecond enrollment image (corresponding to the second distance) todetermine if a match is present. If a match occurs, then authenticationmay proceed.

Variations on these methods are also possible with the system requiringa match at the first distance, but a failure to match at the seconddistance, thereby indicating that the second image is not of atwo-dimensional picture. The processing resulting in a match or failureto match may be any type image or facial recognition processingalgorithm. As with other processing described herein, the processing mayoccur on the mobile device, one or more remote servers, or anycombination of such devices.

All the processing described herein may occur on only the mobile device,only a remote server, or a combination there. The biometric data may bestored on the mobile device or the server or may be split between thetwo for security purposes. For example, the images could be processed onthe mobile device, but compared to enrollment data in the cloud or at aremote server. Or, the images could be sent to the cloud (remote server)for processing and comparison.

Pixel Relative Velocities for Depth Detection

The system may further incorporate the use of an operation referred togenerally herein as pixel velocity analysis to determine whether theauthentication information obtained by the device includes images and/orbiometric information of a live, three-dimensional person. Pixelvelocity analysis tracks the pattern of apparent motion of objects,surfaces, and edges in a visual scene. For instance, pixel velocityanalysis tracks the apparent motion of features in successive imagescaused by the relative motion between a camera and the scene in at leasttwo frames.

In one example, when there is relative movement between the camera andscene, an object closer to the camera within the scene will appear tomove at a different rate than an object farther from the camera. Thismovement or pixel velocity is reflected in changes in pixel values thatrepresent the objects in the picture as the objects change theirlocation in the picture as the camera is moved.

For example, a background element may be represented in the array ofpixels in the images, such as a clock on a wall or a distant tree in thelandscape. A foreground element may also be represented in the array ofpixels as a face or facial feature of a person being imaged. As therelative distance between the camera and the foreground element and thebackground element change, the pixels representing the foreground andthe background elements will change their position within the array ofpixels. Different pixels will represent or capture the particularelements as the camera is moved. Importantly, the differences indistances between the camera and the foreground element and the cameraand the background element cause the rate of change of the pixels (pixelvelocity) of the foreground element and the background element to bedifferent.

In some instances, different features of a foreground object may becompared for relative changes in pixel velocities. In this instance, thepixel velocities of different facial features may be compared, such asthe nose or cheek. The pixel velocities for the different facialfeatures of a real, three-dimensional person will be different based ontheir location on the frame of images and on the relative distances fromthe camera. For example, the pixels that represent the user's nose maymove very little as the camera is moved closer to or further from theuser face while pixels that represent a feature on the outside of theface will move to a greater degree.

In some instances, as the camera is moved closer to the use, somebackground elements will be obscured by the user's face and thus certainelements will be obscured. This would not occur if a two-dimensionalimage of the user was being imaged, such as in a fraud or spoofingattempt, instead of a live three-dimensional user.

The different rates of movement can be captured and compared in framestaken at incremental times. By detecting the different rates of movementor velocities of objects (rate of change over time or based on cameraposition) in the foreground and the background of the image, it can bedetermined that the image is a live, three-dimensional image as comparedto a two-dimensional picture being used in a fraudulent authorizationattempt.

For example, an item in the background (distant from the camera) maymove at a different rate (number of pixel locations in the array perunit of time or per frame) as compared to an item in foreground (closeto the camera). By detecting the differing rates of change in the pixelarray that are represented in the successive images, it can bedetermined that the objects within the images are three-dimensional. Incontrast, if a two-dimensional printed picture is used in an attempt tospoof the system, all the pixels in the images of the printed picturemove at the same rate. This is because all the features shown on atwo-dimensional printed picture are located at essentially the samedistance from the camera.

In particular, items in the foreground move at a higher rate of speed(movement in the pixel array) as compared items in the background. Ifthe device's camera is imaging the user's face, then the foreground willbe the user's face and the background will be the scene behind the user,such as a wall, building, trees, or other items behind the user's face.By detecting the rate of change of pixel locations of items in theforeground as compared to the pixel locations in the pixel array thatforms the image of items in the background, a determination is made thatthe scene is a three-dimensional scene.

Further, pixel velocity analysis also includes edge detection. That,certain identified features in a three-dimensional image may be visiblein some frames, but may not be visible in other frames. This typicallyhappens around the edges of an object in the foreground of an image. Forexample, when a user images his or her face while moving the camera froma distance far away from his or her face to a distance closer to his orher face, objects in the background of an image will disappear as theface of the user takes up more and more space in the frame. Pixelvelocity analysis is configured to identify objects in the backgroundaround the edges of the face and checks that the objects move todisappear behind the face as the face become enlarged, or that thedetected face displaces or covers the features as the face enlarges inthe image as the camera moves closer. This will not occur with atwo-dimensional picture.

Further comparisons can be made by observing the rate of pixel locationswithin the face itself. For example, if pixels on the foreground changeat a different rate as compared to pixels on the background, or displacebackground pixels, then the person can be identified or characterized asthree-dimensional. For example, if pixels showing the nose displacepixels on the upper lip and inner cheeks, and the pixels showing cheeksdisplace pixels representing the ears, and the pixels showing the chindisplace pixels representing the neck in the images as the camera ismoved closer to the face, then the person being imaged can be identifiedas a real three-dimensional person. Further, movement or velocity of theidentified features themselves change based on their location in theframe and their distance from the camera. Thus, features on the outsideof the face exhibit different pixel velocities than features toward thecenter of the face. Similar, features such as the eyes and cheeksexhibit different pixel velocities as compared to the nose due to thedifferent relative distances from the camera.

To perform this comparison, two or more images must be captured. Forexample, a first image is taken at a first distance from the user andthen a second image is taken at a second distance from the user, and soon. The comparison is made between the pixel locations of one or moreelements (items) in the first image and the second image and can becorrelated with the motion of the device as determined by theaccelerometer and gyro.

The pixel velocities detected for various identified features in theimages may be mapped to form a “heat” map or other type mapping of thepixel velocities. In this example where a face is being imaged, detectedpixel velocities are mapped to each of the detected facial features. Forfeatures with high pixel velocities, the map may show “hot” areas, suchas for features on the edge of the face such as a chin or cheekbone. Forareas with low pixel velocities, such as features at the center of theimage like the nose, the map may show “cool” areas. By analyzing thisheat or rate of change map, a determination may be made whether theimage is that of a two-dimensional picture or a live human in athree-dimensional environment.

Pixel velocity analysis may advantageously be used in the abovedescribed authentication system when the device and camera are movedfrom one position to another. For example, the system may conduct pixelvelocity analysis in two or more images as the person authenticatingmoves the device to fit his or her face within the small and large ovalsas shown in FIGS. 13A and 13B.

An example of this process is described with reference to FIG. 14. Whena first image is received by the device or server, feature recognitionis performed on the image to detect predetermined objects within theimage in step 1402. In this instance, facial or feature detection isused to confirm the presence of a user's face and/or facial features onthe user's face, such as the user's nose, eyes, cheekbones, chin, etc.

Next, the system analyses the pixel placement in one or more subsequentframes to determine whether the pixels representing the detectedfeatures correspond with features located in the foreground or thebackground of the scene in step 1404.

In one embodiment, when the user moves the device to fit his or her facewithin the ovals, such as those shown in FIGS. 13A and 13B, the face ofthe user is identified as the foreground of the image, or the featureswithin the ovals 1320, 1330. The area around the face showing the roomor environment of the person is identified as the background of theimage, or the features within area 1315. Additionally, the facialfeatures can be verified to behave with characteristics of relativelydifferent distances and locations in the frame. For example, the nose,mouth, and chin may be considered foreground features while the cheeks,ears and jawline may be considered background features.

In step 1406, the various features are tracked through successive imagesto obtain two-dimensional vectors characterizing the flow or movement ofthe features. The movement of the features in this example is caused asthe user moves the device to fit his/her face within the oval shown inthe exemplary screen displays of FIGS. 13A and 13B. Such movement mayinclude the nose displacing pixels on the upper lip and inner cheeks andthen the cheeks displacing pixels representing the ears and the chindisplacing pixels representing the neck.

The device (processor executing machine readable code stored in memory)then compares image frames (formed by an array of pixels) as the devicemoves closer to the face of the user. The pixels representing objects inthe image are tracked to determine the velocity characteristics of theobjects represented by the pixels in the foreground and the background.The system detects these changes in position of items based on pixeldata, or two-dimensional pixel velocity vectors, by comparing thesuccessive images taken by the device. When the live, three-dimensionaluser is authenticating, velocity characteristics of the foregroundfeatures (face) and the background features differ significantly ascompared to velocity characteristics of a two-dimensional spoof beingimaged. That is, the velocity characteristics of facial features aredifferent for a live, three-dimensional person are different as comparedto a two-dimensional spoof as the user moves the device to fill his/herface in the oval shown in FIGS. 13A and 13B.

Thus, in step 1408, the system checks if the two-dimensional vectors offoreground features match expected values of a live, three-dimensionalperson. The expected values or expected rate of change of an item in animage, defined by pixel location or values, may be based on testing overtime such as expected location, expected displacement, expected rate ofchange of the item, or even expected differences in the rate to changewhich would indicate three-dimensionality (as opposed to a 2D photographor video screen of a person). In this example, testing may set anexpected value of movement or velocities of the ears, cheekbone, nose,etc. When two-dimensional vectors match expected values, the methodproceeds to step 1410 to increase a likelihood that the images are of alive, three-dimensional person. If the two-dimensional vectors do notmatch expected values, (or match values that are expected when atwo-dimensional spoof is used) then the method decreases the likelihoodthat the images are of a live, three-dimensional person as shown in step1412.

When a live, three-dimensional person is being imaged, thetwo-dimensional vectors, or displacement of pixels between successiveimages are different in the foreground and background of the image.Thus, in step 1414, the system also analyzes the two dimensional vectorsof background objects to determine whether these match expected values.The likelihood of the images being of a live, three-dimensional personis again updated in either steps 1410 or 1412.

As explained above, some pixels representing certain background objectsmay appear or disappear completely. For example, as the user moves thedevice from arm's length to closer in towards his or her face, pixels,edges, and/or features of the user's face will have a higher rate ofmovement than features in the background, such as a picture frame on awall, a clock, etc. Additionally, some pixels that are visible on oraround the user's face when the device is furthest out from the userwill no longer be visible when the user moves the device closer to hisor her face. The pixels around a person's face may be defined as thefacial halo and the items in these pixels (facial halo) will no longerbe captured by the camera in the image due to the person's face takingup more of the image and ‘expanding’ due to the movement of the cameracloser to the person's face. As mentioned above, this check may bereferred to as edge detection. In step 1416, the system verifies whetherbackground images around the edges of foreground images match expectedvalues. The system also ensures that pixels representing the edge of theforeground object (such as the face) replace pixels of backgroundobjects near the edges of the foreground object. The likelihood of theimages being of a live, three-dimensional user is adjusted in step 1410and 1412 based on the outcome of the edge detection in step 1416. Thus,by tracking these pixels and the displacement, the system can verifywhether the pixel velocity analysis is consistent with three dimensionalobjects having a foreground and background.

In step 1418, the liveness or three-dimensionality of the user beingimaged and authenticated is validated based on the various checksdescribed above. A determination that the user attempting authenticateis a live person is one element that must be met as part of theauthentication. Thus, attempts at fraudulent access to an account ordevice using screens or photos of the person can be more reliablyprevented. This prevents attempts at fooling the authentication systemwith a two-dimensional image such as a printed picture, a digital aprojection or a digital screen image of a person.

Further enhancements may also be achieved using pixel velocity analysisfor liveness or three-dimensionality. When the user brings the device(camera) closer to the user's face, the facial features will distortdifferently due to the large relative distances between the variousfeatures and the camera and the placement of the features in the fieldof view of the camera as the camera comes closer to the face. Thiseffect may be referred to as perspective distortion. When thisdistortion begins to occur, pixels in the center of the frame thatrepresent the features in the center of the face such as the nose willhave the least amount of distortion in the frame, whereas the pixelsthat represent the outer portions of the face such as the cheeks, thechin, and the forehead will show the most relative pixel movement (morethan pixels at the center of the frame) and the highest acceleration.Thus, the three-dimensionality can also be shown by comparing thefeatures on the face itself. This is because at close proximity to thedevice, facial features closer to the device can be consideredforeground features, and facial features farther from the device arebackground features. For example, pixels representing the nose will showless movement between frames than pixels representing the cheekbonebecause of the nose's shorter relative distance from the camera when thedevice is held at eye level.

Pixel velocity analysis may also be used to track livenesscharacteristics that are very difficult to recreate during a fraudulentauthentication event. For example, the human eyes are never completelystill even when focusing on an object. There is always, quickinvoluntary movement of the eyes as the eyes scan an object, movingaround to locate interesting parts of the object, and developing amental, three-dimensional “map” corresponding to the scene. Thesemovements are called saccades and are involuntary. Saccades last from 20ms-200 ms and serve as the mechanism of eye fixation. Two-dimensionalvelocity vectors, based on movement of the eyes based on pixel values,may thus be generated by the saccadic motion of the eyes across frames.The presence of these vectors, the hertz of the eye jitter and theacceleration of the pixel movement between frames can be compared tomeasurements of verified sessions and can be used to increase confidencethat the user in front of the camera is not an inanimate spoof such as aphoto, a wax sculpture, or doll.

In another example, when a bright light is presented to the human eyes,the pupil will constrict to mitigate the light's path to the retina.Cameras on typical mobile devices such as smart phones generally operateat high enough resolutions that two-dimensional velocity vectors willtrack the pupils constricting when compared over a series of frameswhere the amount of light entering the eyes increases, such as when theuser moves the device and screen closer to his or her face, or when afront-facing flash of a mobile device is activated.

Another feature that may be detected by pixel velocity analysis isreflection off the eye of the user. The surface of the eye reflects alarger amount of the light hitting it when the pupil contracts,providing a brighter reflection of the light emitting object. In thecase of the device with an illuminated screen being moved closer to theface of the user, the size and brightness of the reflection of thedevice's screen will increase while the size of the pupil contracts. Itis possible to observe and document these two-dimensional vectors in aconsistent motion path and then provide a liveness evaluation on videoframe sessions based on the expected two-dimensional vectors beingobserved or absent.

Facial recognition algorithms use landmarked points on the face tomeasure the distance and angles between the facial features. Thiscreates the unique look of individuals and the corresponding uniquebiometric data. In some embodiments, pixel velocity analysis may be usednot only to verify the three-dimensionality of the person, but may alsobe used as an additional or alternative facial recognition algorithm.

In this instance, the device may recognize two-dimensional vectors ofthe features throughout the user's face as the user provides enrollmentimages while moving the camera in and out to fit the ovals as shown inFIGS. 13A and 13B. These two-dimensional vectors formed are caused bythe distortion and movement of the facial features that occurs when thecamera is brought close to the user's face due to the three-dimensionalcharacteristics of the user's face. Tracking the two-dimensional vectorsand mapping the two-dimensional vectors to each of the features of theuser's face results in the creation of a unique “heat” map of the user'sface using a two-dimensional camera. The “heat” map indicates “hotspots” where larger two-dimensional vectors show increased pixelvelocities of certain facial features as opposed to “cool” areas wheresmaller two-dimensional vectors show small pixel velocities of otherfacial features.

Such maps have been found to be unique to each user as the user movesthe device with the camera in and out to fit the ovals as shown in FIGS.13A and 13B. Thus, the map of the two-dimensional vectors of pixelmovement corresponding to facial features resulting from the devicemovement or the head movement can be used itself as biometric data inaddition to verifying three-dimensionality of the person being imaged.That is, the two-dimensional vectors of the various facial features maynot only be compared to expected values to determine thethree-dimensionality of the user's face, but the two-dimensional vectorsof the various facial features created as the user moves the device andcamera relative to the user's face are themselves unique to each userbased on the unique face of the user. Thus, unique heat maps arecaptured based on the three-dimensional facial features of the userregistering with the authentication system, which can be associated withthe user as biometric information for authentication.

Using previously authenticated and stored heat maps of the users face,the system may thus analyze new authentication images with pixelvelocity analysis not only to determine the liveness orthree-dimensionality of the person, but also to authenticate theidentity of the person. These checks may occur simultaneously. That is,the system compares the heat map obtained from the authentication imagesto determine whether it matches the heat map obtained during enrollmentbased on person's unique, three-dimensional facial features. Thetwo-dimensional vectors generated by the pixel velocity analysis arealso examined to ensure they correspond with a three-dimensional personas opposed to a two-dimensional spoof. If the images captured by thedevice camera are determined to not represent a live user or do notmatch the authentication information, then it is contemplated thatauthentication will fail and access will be denied.

Pixel velocity analysis may allow for use of the authentication systemwith a stationary two-dimensional camera, such as on a laptop, an ATM, acar dashboard, or a desktop. That is, with the liveness detectionprovided by the pixel velocity analysis, it may be possible to omit themovement information provided by the accelerometer, magnetometer, andgyroscope of the device, which are lacking in such stationary devices asan ATM machine. This may be done by the user moving his or her head inrelation to a stationary camera. The user's movement may be from normal,observable human movements, or may be a user action that is deliberatelyrequested such as instructing a user to fit his/her face into an ovalchanging size on a screen so that the user leans in to fill the largeroval. Alternatively, the instruction could be a voice instruction. Thisprovides the ability to verify the three-dimensionality of the user fromthe stationary two-dimensional camera. The individual's identity may beverified using facial recognition algorithms while additional identityconfirmation, liveness and three-dimensionality detection can beprovided by the pixel velocity analysis when compared to previouslycaptured data from a similar motion scenario.

Touch Screen Enhancements

Additional added security modifications may include information about auser's finger. Many mobile devices with touch screens can detect thelocation and approximate size of a user's touch on the screen.Accordingly, an approximate size of a user's finger or thumb may bemeasured by the system. In addition to the size of a finger, anorientation angle of the finger or whether the fingers or thumbs of theright or left hand are used can be detected.

In one embodiment, a user selects an account to open, begins enrollmentimaging, or begins authentication imaging by touching the touchscreen ofthe user device. The authentication system may thus detect whether thetouch by a user during authentication corresponds with previously storedenrollment information including the size of the user's finger or thumb,amount of pressure applied to the screen and whether the user is rightor left handed. This adds an additional security layer for theauthentication system.

Furthermore, the authentication system may require that the userinitiates an authentication by touching a fingerprint reader or thetouchscreen in one or more predetermined manners. In one embodiment, asshown in FIG. 15, a touchscreen 1410 may be divided up intopredetermined regions 1420. For example, there may be nine equal,circular, square, or other shaped regions 1420 on the touchscreen 1410of the mobile device. During enrollment, the user selects one of theregions 1420 of the screen 1410 to touch to initiate authentication.During authentication, if the preselected region 1420 is not touched tobegin authentication or during the entire authentication process, thenauthentication is denied. This is but one possible design possibilityand other design options are contemplated.

The regions 1420 on the touchscreen may be visually represented by agrid, or may not be displayed at all on the touchscreen 1410. As shownin FIG. 16, in addition to or in place of the regions 1420, buttons 1520may be displayed on a touchscreen 1510. Here, the user may initiate theauthentication by pressing one or more of the buttons 1520 in apredetermined pattern. The user may also initiate authentication via apredetermined swiped pattern. The position to be touched by the user maychange with each authentication attempt and may be conveyed to the userthrough any instructions from the authentication server, such as a code,number, letter, color, captcha or other indicator.

Voice Parameters

It is also contemplated that the user could record their voice byspeaking a phrase while recording their images during the enrollmentprocess when first using the system. Then, to authenticate, the userwould also have to also speak the phrase when also moving the mobiledevice to capture the image of their face. Thus, one additional pathparameter may be the user's spoken voice and use of voice recognition asanother layer or element of the authentication process.

Image Quality Assurance

The authentication system may also process the images received from themobile device to determine if the images are of sufficient quality. Forexample, the system may check the images for blurriness caused by theimages being out of focus or by the camera lens being obscured byfingerprints, oils, etc. The system may alert that user that the qualityof the images is insufficient (or too bright or too dark) and direct theuser to adjust a focus, exposure, or other parameter, or to clean thelens of the camera.

Autofocus

The authentication system may also utilize an autofocus feature when themobile device camera is equipped with such. For example, when an actual,three-dimensional person is being imaged, the system checks to ensurethat the sharpness of the image changes throughout as the camera performauto-focusing. In another embodiment, the system may control theautofocus so that the camera focuses on a first location or distance tocheck for sharpness (in focus) of a portion of the image containing aface. The system then controls the camera to focus at a second locationor distance where the presence of a face is not detected and check forsharpness (in focus) of a portion of the image. If a three-dimensionalperson in a real environment is being imaged, it is expected that thefocus settings should be different at the first and second locations,which suggests a real person is presently being imaged. However, if theface appears to become large but the focus points of both locations arethe same, this indicates that a two-dimensional video screen is beingimaged, indicating a fraudulent login attempt.

The system may also control the auto-focus of the device to check fordifferent focus points of different particular features in the image.For example, when a person's face is imaged from the front, a person'sear is expected to have a different focus point (more distant) than thetip of a person's nose.

Images of Login Attempt

The authentication server may also be configured to store theauthentication images for a predetermined length of time. The images mayprovide additional security benefits as evidence of a person attemptingto log in to a user's account. For example, the system may store apredetermined number of prior log in attempts, such as twenty loginattempts, or store images from login attempts for a predetermined timeperiod, such as during the past seven days or weeks. Any fraud orattempted fraud will result in pictures of the person attempting thelogin being stored or sent to the authentication server of the accountserver.

The mere knowledge that photos will be taken and sent is a significantdeterrent to any potentially dishonest person because they know theirpicture will be taken and stored, and it is an assurance of security tothe user. Likewise, any attempted and failed attempt can have the photostored and indicator of who is attempting to access the account. It isalso contemplated that an email or text message along with the pictureof the person attempting the failed log in may be sent to the authorizeduser so they know who is attempting to access their account. Thisestablishes the first line of security for the account as the user withthe photo or image also being possessed by the authentication server.

Adaptive Match Thresholds

Further, the level or percentage of correspondence between theenrollment information and the authentication information toauthenticate the user may change over time. In other words, the systemmay comprise an adaptive threshold.

After a user regularly uses the authentication system described above,the user will have logged in with the system by moving the mobile devicein the predetermined path relative to his or her head many times.Accordingly, it may be expected that as the user will gain experienceusing the authentication system, and that the user will gradually settleinto a comfortable and standardized motion path. In contrast, theinitial enrollment movement of a user will likely be the most awkwardand clumsy movement as the user has little experience with theauthentication system.

To make the authentication system more convenient for the user withoutlosing security, the adaptive threshold system allows the enrollmentmovement to adapt so that the user is not locked into the awkward andclumsy initial movement as the enrollment movement. To facilitate this,upon each successfully authorization, the successful authorizationmovement is stored, and the motion path is added to a list of acceptablemotion paths. The list of acceptable motion paths may be limited to apredetermined number of paths. When a new successfully authorization iscompleted and the list of acceptable motion paths is full, the olderenrollment motion path is deleted and the newest is stored in its place.Alternatively, the motion path that is least like the other motion pathsstored on the list may be deleted. Thus, by storing the most alike ornewest motion paths, the enrollment movement may slowly adapt over timeas the user because familiar with the system and settles into acomfortable motion path for authentication.

In addition, other enrollment information may adaptively change in asimilar manner as the user information. For example, successfulauthentication photos or biometric information can be stored as part ofthe enrollment information and old enrollment information may bediscarded over time. In this manner, the authentication system can beconvenient for a user even over a long period of time as the userexperiences aging, facial hair growth, different styles of makeup, newglasses, or other subtle face alterations.

Determining how much variance is allowed over time in the motion path,the biometric information, or both may be set by the entity requiringauthentication to meet that entity's security requirements. Time ornumber of scans after the initial enrollment can be used to modify theadaptive threshold. For example, during a first few days afterenrollment, the threshold may be lower while a security threat is lowand the differences in paths are likely to be higher. After severalauthentications or several days, the threshold may increase. Thethreshold further may be set based on trending data of either the motionpath or biometric information. For example, the threshold may be morelenient in a direction the data is trending, while having a tightertolerance for data against the trend.

A temporal aspect may also be added along with the location information.For example, if the user conducts and authenticates a transaction nearhis home, and then one hour later another transaction is attempted in aforeign country, the transaction may be denied. Or it may be denied ifthe distance between the prior authentication location and the nextauthentication location cannot be traveled or is unlikely to have beentraveled in the amount of time between login or authentication attempts.For example, if the user authenticates in Denver, but an hour later anattempt is made in New York, Russia or Africa, then either first orsecond attempt is fraudulent because the user likely cannot travelbetween these locations in 1 hour.

Further, if the next transaction is attempted at a more reasonable timeand distance away from the first transaction, the level ofcorrespondence threshold may be raised to provide added security,without automatically denying the transaction. Likewise, an altimetermay be used such that if the altitude determined by the mobile device isdifferent than the altitude of the city in which the user is reported tobe located, then this may indicate a fraud attempt. Thus, altitude orbarometric readings from the mobile device may be used to verifylocation and can be cross referenced against GPS data, IP address orrouter location data, or user identified location.

Upgrading Facial Recognition Algorithms

Often, new facial recognition algorithms or other features of anapplication may need to be updated to ensure the security of theauthentication system. When new algorithms are implemented, it isnecessary to obtain new enrollment information corresponding to the newalgorithm. However, it is undesirable to require users who have alreadyauthenticated with the system enroll again every time the application isupdated. This would inconvenience and frustrate the user if, forexample, the user is logging into a payment system to make a purchase ata store, and the application prompts the user to enter a password and/orre-enroll rather than allowing the user to quickly complete thetransaction as planned.

Accordingly, in one embodiment the system performs a biometrics“handoff” to update the enrollment information with a new facialrecognition algorithm based on an application or software update. Forexample, when the software or application is updated with a new facialrecognition algorithm, the application retains the prior facialrecognition algorithm. During the next login attempt the images capturedare used to authenticate the user along with any and all liveness checksusing the older facial recognition algorithm. If the person isauthenticated, the images are then authorized to be used by the newfacial recognition algorithm to generate new enrollment information withthe new biometrics algorithm. The new enrollment biometric informationis considered trustworthy because it is based on a successful loginattempt using the prior biometrics algorithm. This process may be done acertain number of times (login with old algorithm creating enrollmentinformation with new algorithm) until a sufficient biometric profile onthe new facial recognition algorithm is created. Once the new profile iscreated, the prior biometric profile based on the old facial recognitionalgorithm is deleted. In this manner, it is not necessary for a user tore-enroll when the application is updated with new facial recognitionalgorithms or other features.

Random Image Distortion

To provide an additional layer of security to the facial recognitionauthentication system, the system may utilize random image distortion.For example, a user may be assigned a random distortion algorithm uponenrollment into the system. The distortion algorithm may include suchdistortions to the image as widening or narrowing the person's face by apredetermined amount, adding or superimposing a predetermined shape at apredetermined position on the user's face. As one example of this, thedistortion may be a circle superimposed at 100 pixels above the user'sleft eye.

With the uniquely assigned distortion on the images from the user, thebiometric data for that user will be unique to the account or deviceused by the user. That is, the enrollment biometrics stored on theauthentication server or on the mobile device will reflect not only thefacial features of the user, but also will reflect the uniquely assignedimage distortion. Thus, even if an accurate, fraudulent representationof a person were used on a different device or via a different account,the proffered authentication biometrics would not sufficientlycorrespond due to a different or an absence of the unique distortion.Thus, the overall security may be enhanced.

Security Layers

It is noted that each of the above embodiments, modifications, andenhancements may be combined in any combination as necessary to createmultiple layers of security for authentication. For example, the facialrecognition may be combined with motion detection or path detection, ormay operate independently of these features for authentication. Further,when more than one of the above described enhancements or modificationsare combined, the authentication system may be configured so as not toprovide any feedback or indication on which layer failed authentication.

For example, when a predetermined touch pattern to initiateauthentication is combined with the authentication movement and facialauthentication, the system does not indicate whether a touch pattern wasincorrect, or the authentication movement or authentication imagesfailed to correspond to the enrollment information. Instead, the systemprovides an identical denial of authentication no matter what failureoccurs. This is the case when any number of the security featuresdescribed above are combined. In this manner, it is difficult for afraudster to detect what aspect of the fraudulent credentials must becorrected, further enhancing the security of the system.

All of the above features may be incorporated together, or only somefeatures may be used and others omitted. For example, when the deviceprompts the user to move the device so that the user places his or herhead within a first small frame (such as an oval) then to a second largeframe (such as in FIGS. 7A, 7B, 13A, and 13B), the system may beconfigured such that facial recognition need not be performed on theimage(s) in the first frame (distantly captured frames). The security ofthe system is maintained by performing facial recognition throughout theimaging at some point between the first and second frames, and at thesecond frame. This may especially be true when also integrated withanother layer of security, such as checking eye tracking following amoving object on the screen or reading a reflection of a QR code orrandom shape off the user's eye. In another embodiment, when two or morecameras are used creating three dimensional, stereoscopic images, thefacial recognition may not be performed at the first, far away frame,but instead the liveness of the person may be validated at the closer inframe only after the movement of the device. In still other embodiments,other security layers may be used, and the motion parameters may beomitted. Such combinations may be beneficial for larger or stationarydevices, such as gaming laptop computers, personal desktop computers, astationary kiosk, or the like.

Example Applications

Likewise, although described herein as financial account authentication,the authentication using path parameters and image data may beimplemented in any environment requiring verification of the user'sidentity before allowing access, such as auto access, room access,computer access, web site or data access, phone use, computer use,package receipt, event access, ticketing, courtroom access, airportsecurity, retail sales transaction, IoT access, or any other type ofsituation.

For example, an embodiment will be described where the aboveauthentication system is used to securely conduct a retail salestransaction. In this embodiment, a user is enrolled with theauthentication server or an authentication application on the mobiledevice as described above and has generated enrollment informationincluding enrollment images and/or biometrics, and enrollment movement.In this example, the user initiates or attempts to complete atransaction at a retail establishment with a credit card, smart card, orusing a smart phone with NFC capabilities.

The user begins the transaction by swiping a credit card, smart card, orusing an application on a smartphone with NFC capabilities to pay forgoods or services. The retail establishment would then authorize thecard or account with the relevant network of the financial institution(“Gateway”). For example, the retail establishment, through a Gatewaysuch as one operated by VISA or AMERICAN EXPRESS would determine whetherthe account is available and has sufficient available funds.

The Gateway would then communicate with the authorization server toauthorize the transaction by verifying the identity of the user. Forexample, the Gateway may send an authorization request to theauthentication server, and the authentication server then sends anotification, such as a push notification, to the user's mobile deviceto request that the user authenticate the transaction.

Upon receipt of the notification from the authentication server, such asthrough a vibration, beep, or other sound on the mobile device, the usermay then authenticate his or her identify with the mobile device. Theauthentication server may also send information concerning thetransaction to the user for verification by the user. For example, theauthentication server may send information that causes the mobile deviceto display the merchant, merchant location, and the purchase total forthe transaction.

Next, as before, the user may hold the mobile device and obtain aplurality of authentication images as the user moves the mobile deviceto different positions relative to the user's head. While moving themobile device to obtain the authentication images, the mobile phonefurther tracks the path parameters (authentication movement) of themobile device via the gyroscope, magnetometer, and the accelerometer toobtain the authentication movement of the device. The mobile device maythen send the device information, the authentication images, and theauthentication movement to the authentication server. In otherembodiments, the mobile device may process the images to obtainbiometric data and send the biometric data to the server. In still otherembodiments, the mobile device may process the images, obtain theauthentication information, compare the authentication information toenrollment information stored on the mobile device, and send pass/failresults of the comparison to the authentication server.

The authentication server may then authenticate the identity of the userand confirm that the user wishes to authorize the transaction on his orher account if the device information, authentication images and/orbiometrics, and authentication movement correspond with the enrollmentdevice information, the enrollment images and/or biometrics, and theenrollment movement. The authentication server then transmits anauthorization message to the Gateway. Once the gateway has receivedconfirmation of the authorization, the Gateway then communicates withthe retail establishment to allow the retail transaction.

Several advantages may be obtained when a retail transaction isauthorized utilizing the above system and method. Because the identityverification of the user and the confirmation of the transaction iscompleted via the authentication system and mobile device, there is nolonger a requirement for a user to provide his or her credit card orsignature, or to enter a pin number into the retailer's point of salesystem. Further, the retail establishment does not need to check a photoidentification of the user. The above method and system also has theadvantage that it provides secure transactions that can work with mobileand online transactions that do not have cameras, such as securitycameras, on the premises.

In the secure retail transaction described above, the user obtains thetotal amount due on his or her mobile device from the retailestablishment via the Gateway and authentication server. However, in oneembodiment, the mobile phone may use the camera as a bar code, QR code,or similar scanner to identify the items and the prices of the itemsbeing purchased. The mobile device may then total the amount due and actas the checkout to complete the transaction with the retailestablishment.

In another embodiment, a user of the application may want to anonymouslypay an individual or a merchant. In this instance, the user woulddesignate an amount to be paid into an application, and the applicationwould create a unique identifying transaction number. This number maythen be shown to the second user, so the second user can type theidentifying transaction number on an application on a separate device.The unique identifying transaction number may also be sent from the userto the second user via NFC, Bluetooth, a QR code, or other suitablemethods. The second user may also type the amount and request payment.

Upon receiving the payment request and unique identifying transactionnumber, the authentication server may send a notification to the firstuser's mobile device to authenticate the transaction. The user wouldthen verify his or her identity using the facial recognitionauthentication system described above. The user may alternatively oradditionally verify his or her identity using other biometric data suchas a fingerprint or retina scan, path-based motion and imaging, or theuser may enter a password. Upon authentication, the user's device wouldsend a request to the user's payment provider to request and authorizepayment to the second user. In this manner, the payment may be donesecurely while the users in the transaction are anonymous.

According to one embodiment, as an additional measure of security, theGPS information from the mobile device may also be sent to theauthentication server to authenticate and allow the retail transaction.For example, the GPS coordinates from the mobile device may be comparedwith the coordinates of the retail establishment to confirm that theuser is actually present in the retail establishment. In this manner, acriminal that has stolen a credit card and attempts to use the card froma distant location (as compared to the retail location) is unable tocomplete a transaction because the user's phone is not at the locationof the retail establishment. IP addresses may also be used to determinelocation.

As explained above, the level or percentage of correspondence betweenthe enrollment information and the authentication information toauthenticate the user may also be adjusted based on the coordinates ofthe GPS of the mobile device. For example, if the retail establishmentand GPS coordinates of the mobile device are near a user's home, thenthe level of correspondence may be set at a lower threshold, such as ata 99% match rate. Alternatively, if the location is very far from theuser's home, and is in a foreign country, for example, then the level ofcorrespondence may be set at a higher threshold, such as at a 99.999%match rate.

While various embodiments of the invention have been described, it willbe apparent to those of ordinary skill in the art that many moreembodiments and implementations are possible that are within the scopeof this invention. In addition, the various features, elements, andembodiments described herein may be claimed or combined in anycombination or arrangement.

What is claimed is:
 1. A method for authenticating a user via a user'scamera equipped computing device, the method, during an authenticationsession comprising: capturing at least one first image of the user takenwith the camera of the computing device at a first distance from theuser and processing said at least one first image to obtain firstbiometric data based on facial recognition data from the at least onefirst image; moving the camera from a first location to a secondlocation or the user moving from the first location to the secondlocation to change the distance between the user and the camera from thefirst distance to a second distance; capturing at least one second imageof the user taken with the camera of the computing device at the seconddistance from the user, the second distance being different than thefirst distance, and processing the at least one second image to obtainsecond biometric data based on facial recognition data from the at leastone second image; comparing the first biometric data with first storedbiometric data to determine whether the first biometric data matches thefirst stored biometric data; comparing the second biometric data withsecond stored biometric data to determine whether the second biometricdata matches the second stored biometric data, the second storedbiometric data being distinct from the first stored biometric data; andauthenticating the user when: the first biometric data matches the firststored biometric data and second biometric data matches the secondstored biometric data; and differences between the at least one firstimage and the at least one second image match expected differencesresulting from movement of the camera from the first location to thesecond location or movement of the user from the first location to thesecond location, which causes the change in distance between the userand the camera.
 2. The method according to claim 1, further comprising:interpolating the first stored biometric data and the second storedbiometric data to obtain estimated intermediate biometric data;capturing at least one third image of the user taken with the camera ofthe computing device at a third distance from the user, the thirddistance being between the first and the second distances, andprocessing the at least one third image to obtain third biometric databased on facial recognition data from the at least one third image; andcomparing the estimated intermediate biometric data with the thirdbiometric data to determine whether the third biometric data matches theestimated intermediate biometric data.
 3. The method according to claim1, further comprising verifying the presence of the user's ears in theat least one first image, and verifying the absence or reducedvisibility of the user's ears in the at least one second image, whereinthe first distance is larger than the second distance.
 4. The methodaccording to claim 1, wherein the computing device is configured todisplay one or more prompts on a screen of the computing device to guidethe user to capture the at least one first image and the at least oneOff second image at the first and second distances.
 5. The methodaccording to claim 4, wherein the one or more prompts are ovals sized onthe screen within which the face of the user is placed to capture the atleast one first image and the at least one second image at the first andsecond distances.
 6. The method according to claim 4, wherein thecomputing device is a hand-held device, and the user holds the device atthe first and second distances to capture the at least one first and theat least one second images.
 7. The method according to claim 6, whereinthe hand-held device comprises at least one of gyroscope, accelerometer,and a magnetometer, and the hand-held device captures movementinformation of the device as the hand-held device is moved from thefirst distance to the second distance.
 8. The method according to claim1, further comprising displaying an image on a screen of the computingdevice while capturing the at least one first and/or the at least onesecond image, and processing the at least one first image and/or the atleast one second image to detect a reflection of the displayed image offof the user's face.
 9. The method according to claim 8, wherein theimage is a randomly determined color that is reflected off of the user'sface or eye.
 10. The method according to claim 8, wherein the image is acoded image that is reflected off of the user's eye.
 11. The methodaccording to claim 10, wherein the image is a bar code, and a reflectionof the bar code off of the user's eye is detected in the at least onefirst image and/or the at least one second image.
 12. The methodaccording to claim 1, wherein the first biometric data, the secondbiometric data, the first stored biometric data, and the second storedbiometric data are maintained on the computing device.
 13. A method forauthenticating a user via a user's camera equipped computing device, themethod, during an authentication session comprising: capturing at leastone first image of the user taken with the camera of the computingdevice at a first distance from the user and processing said at leastone first image to obtain first biometric data based on facialrecognition data from the at least one first image; capturing at leastone second image of the user taken with the camera of the computingdevice at a second distance from the user, the second distance beingdifferent than the first distance, and processing the at least onesecond image to obtain second biometric data based on facial recognitiondata from the at least one second image; comparing the first biometricdata with first stored biometric data to determine whether the firstbiometric data matches the first stored biometric data; comparing thesecond biometric data with second stored biometric data to determinewhether the second biometric data matches the second stored biometricdata, the second stored biometric data being distinct from the firststored biometric data; and authenticating the user when: the firstbiometric data matches the first stored biometric data and secondbiometric data matches the second stored biometric data; and the firstbiometric data does not match the second biometric data due to thechange in distance between the user and the camera when capturing the atleast one first image, at the first distance, and the at least onesecond image, at the second distance; wherein the computing device isconfigured to display one or more prompts on a screen of the computingdevice to guide the user and frame the user's face within the promptsduring capture of the at least one first image at the first distance andthe at least one second image at the second distance.